Protein Autoregressive Modeling via Multiscale Structure Generation

🎯 Pitch

A single pretrained PAR model generates protein backbones and perform zero-shot motif scaffolding and prompt-based design—tasks that normally require separate, fine-tuned models—by autoregressively predicting structure from coarse to fine scales. It also achieves a 2.5× sampling speedup by dynamically switching from stochastic to deterministic sampling partway through generation, without quality loss.

1. Executive Summary

This paper introduces Protein Autoregressive Modeling (PAR), the first multi-scale autoregressive framework for protein backbone generation that generates Cα coordinates via coarse-to-fine next-scale prediction — avoiding the discretization and unidirectional-dependency limitations of prior AR protein models. PAR combines a non-equivariant autoregressive transformer for scale-wise conditioning with a flow-based backbone decoder, and mitigates exposure bias through noisy context learning and scheduled sampling (training on corrupted prior-scale inputs and substituting model predictions for ground-truth context during training). On unconditional generation, PAR achieves an FPSD of 161.0 against the PDB dataset with favorable scaling behavior, enables a 2.5× sampling speedup by orchestrating SDE sampling at coarse scales with ODE sampling at fine scales, and demonstrates zero-shot generalization to prompt-based generation (specifying coarse layouts with as few as 16 points) and motif scaffolding without any fine-tuning — establishing that multi-scale autoregressive modeling captures protein distributions competitively with diffusion methods while unlocking flexible conditional generation that single-scale approaches require additional training to achieve.

2. Context and Motivation

The Core Problem: Autoregressive Modeling Has Largely Bypassed Protein Structure Generation

The central question this paper tackles is: can autoregressive (AR) modeling — the paradigm that powers modern large language models — be applied effectively to protein backbone generation? This matters because AR models have demonstrated two properties that are highly desirable in any generative domain: scalability, where performance predictably improves with model size and training compute (Kaplan et al., 2020), and zero-shot generalization, where a single pretrained model can perform diverse tasks without task-specific fine-tuning (Brown et al., 2020). If these properties could be transferred to protein design, it would unlock flexible, general-purpose generative models that can handle a wide range of design tasks without the engineering overhead of retraining for each new conditioning signal.

Despite AR models dominating language modeling and more recently achieving state-of-the-art results in image generation (Tian et al., 2024), protein backbone generation has remained largely in the domain of diffusion models and flow matching approaches (Yim et al., 2023a; Watson et al., 2023; Geffner et al., 2025). The paper explicitly identifies the near-total absence of AR modeling in this space, noting that the only prior attempt (Gaujac et al., 2024) modeled discretized structure tokens with a causal transformer — an approach the authors argue is fundamentally limited. This gap is the motivating tension: AR models offer capabilities that diffusion models have not demonstrated (particularly zero-shot task flexibility), but no one has yet built an AR framework that respects the structural properties of proteins.

Why This Problem Is Important

The paper's framing implies several dimensions of practical and theoretical significance:

Practical protein design workflows are inherently conditional. Real-world protein design problems rarely take the form of "generate a random plausible backbone." Instead, researchers want to specify constraints: a particular binding pocket must be preserved (motif scaffolding), a general fold topology must be maintained, or a structural scaffold must adapt to a new functional site. Diffusion models can handle these tasks, but typically require task-specific training — conditioning signals (masks, motif coordinates, structural constraints) must be incorporated during training, and the model loses flexibility when asked to handle a conditioning type it wasn't trained for. An AR model with zero-shot generalization, by contrast, could handle novel conditioning signals at inference time simply by modifying the input context, similar to how LLMs perform few-shot learning by conditioning on in-context examples. PAR's ability to do prompt-based generation with 16 points and motif scaffolding without fine-tuning (Figures 3 and 4) directly addresses this workflow bottleneck.

The protein structure distribution is hierarchical by nature. Proteins span multiple scales of structural organization: tertiary fold topology (global architecture), secondary structure elements (local α-helices and β-sheets), and atomic coordinates (the finest detail). A generative model that leverages this hierarchy rather than ignoring it should — in principle — produce more coherent structures, because the coarse layout constrains the space of plausible fine details. The paper positions its multi-scale AR framework as naturally aligned with this biological reality, whereas single-scale diffusion models generate all residues simultaneously without explicit hierarchical structure. This is not merely aesthetic: the paper argues that establishing a reliable coarse topology first enables more efficient refinement at fine scales, which directly leads to the 2.5× sampling speedup demonstrated in Table 2.

The training-inference mismatch (exposure bias) is an under-addressed challenge for structured generation. In standard AR training with teacher forcing, the model sees ground-truth prior tokens as context. At inference, it sees its own previously-generated (and potentially erroneous) outputs. This mismatch causes error accumulation across autoregressive steps. While exposure bias is well-studied in language generation (Arora et al., 2022), the paper identifies it as substantially more severe in protein backbone generation because structural errors compound geometrically — a small misplacement at a coarse scale cascades into large-scale structural violations at fine scales. The authors' preliminary study found that naive teacher forcing "greatly reduces the designability of generated structures" (Section 3.3). Addressing this became not just a refinement but a prerequisite for making the multi-scale AR approach viable at all.

Where Prior Approaches Fall Short

The paper identifies two broad categories of prior work — diffusion/flow models and AR models — and articulates specific limitations of each that PAR is designed to overcome.

Diffusion and Flow Models: Single-Scale, Task-Specific

Most prior protein backbone generators — FrameDiff (Yim et al., 2023b), RFDiffusion (Watson et al., 2023), FrameFlow (Yim et al., 2023a), Genie2 (Lin et al., 2024), Proteina (Geffner et al., 2025) — are diffusion or flow-based models that operate at a single scale. During generation, they sample noise from a prior distribution and iteratively denoise it into a complete backbone structure, treating all residues simultaneously. This approach has proven highly effective at unconditional generation: these models produce designable, diverse structures that match the training distribution in terms of metrics like Fréchet Protein Structure Distance (FPSD) and secondary structure composition.

However, single-scale operation carries specific limitations:

No natural handle for multi-granularity conditioning. Because the generation process does not explicitly separate coarse topology formation from fine-detail refinement, conditioning on a partial specification — say, a 16-point coarse layout or a motif of atomic coordinates — requires explicit integration into the denoising process. This typically means training with those conditioning signals, as Proteina does for motif scaffolding. A model trained with one type of conditioning cannot, in general, generalize to others at test time without retraining.
Fixed computational cost regardless of difficulty. Diffusion models use a fixed number of denoising steps (typically 400–1000 in current protein models) regardless of the structure's complexity. There is no mechanism to spend less computation on regions of the structure that are easier to generate or to adapt the computational budget based on what has been generated so far.
Sampling efficiency is limited by single-scale architectures. The single-scale design forces the model to denoise the entire structure at the full sequence length, where transformer attention costs scale quadratically with sequence length. PAR's multi-scale approach enables the model to establish the global layout at a short, fixed-length coarse scale (64 residues), where attention is cheap, and then progressively upsample to longer sequences where the expensive attention only needs to refine rather than create structure from scratch.

Prior AR Protein Models: Discretization and Unidirectional Dependency

The paper identifies only one prior attempt at AR protein structure generation: Gaujac et al. (2024), which discretizes protein structures into tokens (similar to VQ-VAE tokenization in image generation) and models them with a causal transformer. This approach inherits two fundamental problems from standard AR modeling in continuous domains:

Discretization reduces structural fidelity. When Cα coordinates (continuous 3D values) are mapped to discrete tokens through vector quantization, fine-grained structural details are inevitably lost. The paper cites Hsieh et al. (2025) in noting that discrete structure tokens "often reduce structural fidelity and limit generation quality." This is the same limitation that motivated Li et al. (2024) to develop continuous AR image generation (MAR), which PAR directly builds upon: rather than discretizing the data and predicting tokens, use the AR model to produce a conditioning vector that feeds into a continuous diffusion/flow decoder.

Unidirectional dependency conflicts with protein biochemistry. Standard AR models assume each token depends only on previous tokens — a unidirectional, left-to-right causal structure. But protein residues exhibit strong bidirectional dependencies: two residues that are distant in sequence (e.g., positions 10 and 150) may be spatially adjacent in the folded structure, forming hydrogen bonds or hydrophobic contacts. A causal transformer that generates residues in sequence order sees residue 10 when predicting residue 150, but not vice versa — it cannot model the mutual structural constraint. The paper argues this "limits the quality of previous attempts on autoregressive structure generation" (Section 1).

The authors' key insight is that next-scale prediction elegantly sidesteps both problems. By generating an entire coarse-scale structure (all residues at that scale, simultaneously via flow matching) conditioned on all previous coarser scales, the model captures bidirectional dependencies within each scale while maintaining the autoregressive property across scales. No discretization is needed because the flow decoder models continuous coordinates directly. This framework — motivated by VAR (Tian et al., 2024) in image generation and MAR (Li et al., 2024) for continuous AR modeling — has not previously been applied to protein structures.

How PAR Positions Itself Relative to Existing Work

The paper makes a deliberate conceptual claim: PAR is not merely a competing generative model but a more general framework that subsumes existing flow-based approaches. Section 5 states this explicitly: "PAR provides a more general framework, incorporating flow-based models as a special case when restricted to a single scale." If PAR is configured with only one scale (the full-resolution structure), the AR transformer produces a single conditioning vector, and the flow decoder is trained and sampled exactly like a standard flow matching model. The multi-scale extension adds autoregressive structure without changing the underlying flow machinery — the flow decoder remains compatible with techniques from the flow-matching literature, including self-conditioning and SDE sampling.

This positioning is strategic: it allows PAR to claim the best of both worlds. On unconditional generation benchmarks, PAR is competitive with or exceeds pure flow models (Table 1: 161.0 FPSD vs. PDB after fine-tuning, compared to 271.3 for Proteina at 400M). At the same time, PAR inherits from AR modeling capabilities that single-scale flow models lack: zero-shot conditional generation (prompts, motifs), interpretable coarse-to-fine generation that mirrors the biological hierarchy of protein structure, and the ability to allocate different sampling strategies (SDE vs. ODE) at different scales for efficiency.

The paper also explicitly connects to the scaling laws conversation that has driven LLM development. By demonstrating in Figure 5 that PAR's performance on FPSD and related metrics improves with both model size (60M → 200M → 400M parameters) and training steps (200K → 400K → 600K), the authors position PAR as following the same scaling paradigm that made AR models dominant in language. This is an empirical argument, not just a methodological one: if PAR scales predictably with compute, it suggests that future investment in larger models and datasets will yield proportional returns — a property that, while well-established in NLP, has not been demonstrated for protein AR models before.

Finally, the paper addresses the exposure bias problem head-on, acknowledging it as a barrier that must be overcome for multi-scale AR to be practical. The combination of noisy context learning (training on corrupted prior-scale inputs) and scheduled sampling (occasionally substituting the model's own predictions for ground-truth context during training) is adapted from techniques in language (Bengio et al., 2015) and image AR modeling (Ren et al., 2025). The paper's contribution here is not the techniques themselves but their necessity and effectiveness in the protein domain — Table 3 shows that without these mitigations, sc-RMSD degrades to 2.20, which is substantially worse than the 1.48 achieved with both techniques combined. This establishes that exposure bias is not a minor issue in structured generation but a central challenge that must be solved for multi-scale AR to work at all.

3. Technical Approach

3.1 Reader Orientation

PAR is a multi-scale autoregressive system that generates protein backbones (Cα atom coordinates) by first predicting a coarse structural layout and then progressively refining it through finer scales, analogous to a sculptor blocking out rough forms before adding detail. The system solves the problem of applying autoregressive modeling to protein structures — which have inherently bidirectional residue dependencies and exist in continuous 3D space — by decomposing generation into a hierarchy of scales where each scale is generated conditionally on all previous coarser scales, avoiding both discretization artifacts and unidirectional sequence constraints.

3.2 Big-Picture Architecture (Diagram in Words)

PAR consists of three major components connected in an autoregressive loop:

Multi-scale downsampling (q_decompose) — A deterministic, non-parametric operation that takes a protein backbone x ∈ ℝ^(L×3) and produces a hierarchy of coarser representations {x¹, x², ..., xⁿ = x} by interpolating coordinates along the sequence dimension. This happens once per protein during training to create targets and context; at inference, the hierarchy is built autoregressively.
Autoregressive transformer (T_θ) — A stack of non-equivariant attention layers that takes all previously generated coarser scales (upsampled to the current target size) as input and produces a scale-wise conditioning embedding zⁱ. This embedding encodes structural information from coarser scales into a form the decoder can use.
Flow-based backbone decoder (v_θ) — A shared (same weights across all scales) flow matching network that generates Cα coordinates at scale i by denoising Gaussian noise, conditioned on zⁱ from the AR transformer. At inference, it runs an ODE or SDE to transform noise into structure.

Information flow at inference: The AR transformer produces z¹ from a learned start token → the flow decoder generates coarse structure x¹ from noise conditioned on z¹ → x¹ is upsampled and fed back to the AR transformer to produce z² → the flow decoder generates x² conditioned on z² → repeat n times until the full-resolution backbone xⁿ = x is produced.

Information flow at training: A real protein structure x is downsampled into n scales {x¹, ..., xⁿ}. For each scale i, the AR transformer encodes all previous scales {x¹, ..., xⁱ⁻¹} to produce zⁱ. The flow decoder is trained to denoise xⁱ from Gaussian noise, conditioned on zⁱ. All scales are trained jointly with a single flow matching loss.

3.3 Roadmap for the Deep Dive

First, the formal probabilistic formulation (Equation 1), which defines what "multi-scale autoregressive" means mathematically and establishes the decomposition into downsampling and conditional generation.
Second, the multi-scale downsampling operation (Equation 2), since all subsequent components depend on these hierarchical representations and the choice of scale configurations fundamentally shapes what the model learns.
Third, the autoregressive transformer and flow decoder together (Equations 3–5), because they form the core generative engine — the transformer converts structural history into conditioning, and the decoder converts conditioning plus noise into coordinates.
Fourth, the inference-time generation procedure, including the SDE/ODE sampling formulation (Equation 6), since this is where the training-time components are actually deployed and where the multi-scale orchestration enables efficiency gains.
Fifth, the exposure bias mitigation strategies (noisy context learning and scheduled sampling), because they are not architectural components but training interventions that proved essential for making the multi-scale AR approach work at all.
Sixth, key implementation details (scale embedding, interpolated position encoding, self-conditioning) that are not part of the core formulation but significantly affect practical performance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that protein backbone generation can be formulated as a multi-scale autoregressive process where each scale is generated by a flow matching decoder conditioned on all coarser scales, avoiding both the discretization losses of token-based AR models and the unidirectional dependency constraints that conflict with protein biochemistry.

The Probabilistic Formulation

The paper formalizes the generation of a protein backbone x ∈ ℝ^(L×3) (Cα coordinates for L residues) through an autoregressive decomposition over n scales:

$p_θ(x) = 𝔼_{X \sim q_{decompose}(·|x)}[p_θ(X = {x¹, ..., xⁿ})]$

$= 𝔼_{X \sim q_{decompose}(·|x)} ∏_{i=1}^{n} p_θ(xⁱ | X^{<i})$

where X = {x¹, ..., xⁿ} is a set of n structural representations at increasing resolution, xⁿ = x (the finest scale is the full backbone), X^{<i} = {x¹, ..., xⁱ⁻¹} denotes all scales coarser than i, and q_{decompose}(·|x) is a distribution over possible decompositions of a given backbone into scales.

What it computes: The probability of the full backbone x under PAR is the expected value (over possible decompositions) of the product of per-scale conditional probabilities — each scale i is generated given all coarser scales. This is a standard autoregressive factorization, but applied across scales rather than across sequence positions.

Why this form: Standard AR models factorize across tokens: p(x) = ∏_{t} p(x_t | x_{<t}). For protein backbones, token-level factorization would require either discretizing coordinates (losing fidelity) or imposing a unidirectional residue order (violating bidirectional biochemical dependencies). By factorizing across scales instead, PAR generates an entire structural representation (all residues at that scale simultaneously) conditioned on coarser representations — the within-scale dependencies are captured by the flow decoder (which sees all positions at once), while the across-scale dependencies are captured autoregressively. The expectation over q_{decompose} acknowledges that there may be multiple valid ways to decompose a structure into scales; however, as the paper notes, q_{decompose} is actually made deterministic (a single mapping for each x), which simplifies the expectation away: p_θ(x) = ∏_{i=1}^{n} p_θ(xⁱ | X^{<i}) without marginalization. This is a practical choice — deterministic downsampling makes training simpler and the paper's empirical results show it works well — but the general formulation leaves room for stochastic decompositions in future work.

Multi-Scale Protein Downsampling

The downsampling operation q_{decompose} constructs a hierarchy of representations from a single backbone structure. Given a protein x ∈ ℝ^(L×3), it produces n increasingly fine scales:

$q_{decompose}: x ↦ X = {x¹, x², ..., xⁿ}$

$= \{Down(x, size(1)), Down(x, size(2)), ..., x\}$

where Down(x, size(i)) ∈ ℝ^(size(i)×3) is a downsampling operation that interpolates the coordinate sequence x along its length dimension to produce size(i) 3D centroids, and size(1) < size(2) < ... < size(n) = L.

What it computes: Starting from the full-resolution backbone (e.g., 256 residues × 3 coordinates), the operation applies 1D interpolation along the sequence axis to produce shorter coordinate sequences at each scale. For example, with scales {64, 128, 256}, the 256-residue protein is first interpolated to 128 points, then to 64 points. Each downsampled representation is a "blurred" version of the structure — the 64-point representation captures the global fold topology (where helices and sheets are in 3D space), the 128-point representation adds more local detail, and the 256-point representation is the full atomic-level structure.

Why this form: The paper considered and rejected two alternatives. Discretization (mapping coordinates to tokens) would lose fine structural detail — the paper explicitly notes that prior discrete structure tokenizers "reduce structural fidelity and limit generation quality" (Section 1). Residue-by-residue autoregressive generation would impose a unidirectional order that conflicts with the bidirectional spatial dependencies in proteins (residues far apart in sequence can be adjacent in 3D). Sequence-based downsampling preserves the continuous coordinate representation while restructuring the autoregressive order from "per-residue" to "per-scale." The paper verifies in Section 8.8 (Table 13) that this downsampling preserves pairwise spatial relationships: comparing distance maps computed from downsampled coordinates versus downsampled full-resolution distance maps yields LDDT scores of 1.0 at all tested scales (16, 32, 64, 128), with RMSE values of only 0.362, 0.275, 0.217, and 0.170 respectively. This means the coarse representations faithfully capture the essential spatial organization of the protein, just at reduced resolution.

Scale configuration. The set of sizes S = {size(1), ..., size(n)} can be defined in two ways:

By length: scales are fixed hyperparameters independent of protein length, e.g., S = {64, 128, 256}. If a protein has length L ≤ 128, it would use only the first two autoregressive steps; if L ≤ 64, only the first step. This means shorter proteins require fewer autoregressive iterations.
By ratio: scales adapt to protein length, e.g., S = {L/4, L/2, L}. Every protein uses exactly n autoregressive steps, but the absolute sizes vary.

The paper reports (Table 4) that defining scales by length yields slightly better performance on distributional metrics: a 3-scale {64, 128, 256} configuration achieves FPSD of 279.63 vs. PDB and designability of 85.0%, compared to 310.64 and 86.4% for {L/4, L/2, L}. The default configuration throughout the paper is {64, 128, 256} (3 scales). Increasing to 4 or 5 scales degrades designability (77.8% for {64, 128, 192, 256}), which the authors attribute to error accumulation across more autoregressive steps and increased exposure bias.

Coarse-to-Fine Backbone Autoregressive Modeling

The core generative process models each scale's structure conditioned on all coarser scales. The paper decomposes this into two learned components — an AR transformer that produces conditioning, and a flow decoder that generates coordinates — rewriting the per-scale likelihood as:

$p_θ(X = {x¹, ..., xⁿ}) = ∏_{i=1}^{n} p_θ(xⁱ | X^{<i})$

$= ∏_{i=1}^{n} p_θ(xⁱ | zⁱ = T_θ(X^{<i}))$

where T_θ is the autoregressive transformer that maps prior-scale structural history to a conditioning embedding zⁱ, and p_θ(xⁱ | zⁱ) is the flow-based atomic decoder v_θ that generates coordinates at scale i conditioned on that embedding.

What it computes: For each scale i, the system first encodes all coarser scales into a conditioning vector zⁱ, then uses that vector to guide a flow matching process that transforms Gaussian noise into the structural coordinates xⁱ. The generation at scale i is a continuous denoising process (the flow decoder), but the progression from scale to scale is discrete and autoregressive.

Why this form: This design separates two concerns that standard AR models conflate. The AR transformer handles the scale-to-scale dependency — it sees the coarse structural history and decides what structural information is relevant for the next finer scale. The flow decoder handles the within-scale dependency — it sees all positions at the current scale simultaneously (through its transformer attention) and generates them jointly via flow matching. This decomposition means the AR transformer does not need to generate coordinates directly (avoiding discretization) and the flow decoder does not need to model the autoregressive scale progression (it just needs to denoise given a conditioning vector). The approach is directly inspired by MAR (Li et al., 2024) for image generation, where an AR model produces conditioning for a diffusion network, extended here to the multi-scale protein setting.

Autoregressive Transformer for Scale-Wise Conditioning

The AR transformer T_θ is a standard (non-equivariant) transformer that processes the history of coarser scales to produce a conditioning embedding for the current scale:

$zⁱ = T_θ(X^{<i}) = T_θ([bos, Up(x¹, size(2)), ..., Up(xⁱ⁻¹, size(i))])$

where bos ∈ ℝ^(size(1)×3) is a learnable start-of-sequence embedding (initialized randomly and trained), Up(xʲ, size(i)) upsamples the scale-j structure xʲ to have size(i) points via interpolation, and [·, ·, ..., ·] denotes concatenation along the sequence dimension before feeding into the transformer.

What it computes: At scale i, the AR transformer receives as input a sequence of 3D points formed by concatenating: (1) the learned bos embedding (size size(1)), (2) the first scale's structure upsampled to size size(2), (3) the second scale's structure upsampled to size size(3), and so on, through the (i-1)-th scale upsampled to size size(i). The total sequence length fed to the transformer is size(1) + size(2) + ... + size(i). The transformer processes this sequence and the output embedding corresponding to the final scale segment becomes zⁱ — the conditioning for generating xⁱ.

Concretely, for a 3-scale model {64, 128, 256} generating scale 2 (128 points): the input is [bos (64 points), Up(x¹, 128) (128 points)] concatenated to 192 total points. The transformer output segment corresponding to the last 128 points is extracted as z². For scale 3 (256 points): the input is [bos (64 points), Up(x¹, 128) (128 points), Up(x², 256) (256 points)] concatenated to 448 total points, and the last 256 points' output is z³.

Why this form: This concatenation-based input design means the AR transformer sees all prior scales at their native resolutions (upsampled to match the current target size for dimensional compatibility). The upsampling ensures that coarser scales provide spatial context at the appropriate granularity — a 64-point coarse layout upsampled to 256 points provides a smooth interpolation that suggests where the fine-scale residues should roughly be located. The bos token serves as a learned prior for the very first scale, analogous to the start token in language models. The transformer's self-attention across all positions allows information to flow between scales, which the attention map analysis (Figure 6) confirms: each scale primarily attends to the immediately previous scale but retains non-negligible attention to earlier scales, enabling the model to maintain structural consistency throughout the generation process.

Architectural details. The AR transformer uses the smallest configuration from Proteina (60M parameters): 512-dimensional sequence representations, 128-dimensional conditioning embeddings, 12 attention heads, 12 transformer layers, sinusoidal time encoding dimension 196, and interpolated position encoding dimension 196 (see Table 6). The paper reports in Section 8.7 (Table 12) that scaling the AR transformer from 60M to 400M parameters does not improve performance, and may even hurt due to increased overfitting to ground-truth context (exposure bias). Consequently, all PAR variants use a 60M AR transformer while scaling only the flow decoder.

Position encoding. The AR transformer uses interpolated position encodings: at each scale i, size(i) position indices are uniformly sampled from the interval [1, L] using linspace(1, L, size(i)). This means that a position at scale 1 might have index 1 (the first residue), index 128 (the middle residue), or index 256 (the last residue), while at scale 3 every residue from 1 to 256 gets a unique index. Coarse scales have widely-spaced indices, encouraging the model to capture global structural layout; fine scales have dense indices, encouraging local detail. The position indices are converted to sinusoidal embeddings before being fed to the transformer.

Scale embedding. Because the same flow decoder is shared across all scales, the model needs a way to distinguish which scale it is generating. A learnable scale embedding vector (unique per scale) is concatenated to the conditioning zⁱ before it is injected into the decoder. This allows the decoder to adjust its behavior: at coarse scales, it generates smooth, global layouts; at fine scales, it adds sharp local detail. An ablation in Table 11 shows that removing this scale embedding reduces designability from 96.6% to 92.8% and increases FPSD vs. PDB from 161.0 to 175.1, confirming its importance. However, making the scale embedding learnable fixes the model to the number of scales it was trained with; the paper also experiments with removing it to enable "scale-agnostic" inference at different numbers of scales, though with degraded designability.

Flow-Based Atomic Decoder

The flow-based backbone decoder v_θ generates the continuous Cα coordinates at scale i through flow matching (Lipman et al., 2022). It is a transformer-based network (matching the Proteina architecture) that takes a noised version of the target coordinates, a time variable, and the conditioning embedding, and predicts the velocity field that transports noise to data:

$ℒ(θ) = 𝔼_{x \sim p_D} \left[ \frac{1}{n} \sum_{i=1}^{n} \frac{1}{size(i)} 𝔼_{tⁱ \sim p(tⁱ), εⁱ \sim N(0, I)} \left\| v_θ(xⁱ_{tⁱ}, tⁱ, zⁱ) - (xⁱ - εⁱ) \right\|² \right]$

where x is a training protein from the data distribution p_D, n is the number of scales, size(i) normalizes the loss per scale by the number of residues (so coarser scales with fewer points don't dominate), tⁱ ∈ [0, 1] is a time variable sampled from p(t) (the same t-sampling distribution as in Geffner et al. (2025)), εⁱ ∼ N(0, I) is Gaussian noise, xⁱ_{tⁱ} = tⁱ · xⁱ + (1 - tⁱ) · εⁱ is the interpolated (noised) sample at time tⁱ, zⁱ = T_θ(X^{<i}) is the conditioning from the AR transformer, and v_θ is the vector field network being trained.

What it computes: For each training protein, each scale i, and each sampled time tⁱ, the model takes the interpolated sample xⁱ_{tⁱ} (a mix of the true structure and noise, weighted by tⁱ), the time tⁱ, and the conditioning zⁱ, and predicts the velocity vector v_θ(xⁱ_{tⁱ}, tⁱ, zⁱ). The loss is the mean squared error between this predicted velocity and the true velocity (xⁱ - εⁱ) — the direction from pure noise to clean data. The factor 1/size(i) normalizes per scale so that the loss contribution is per-residue rather than per-scale (coarse scales have fewer residues and would otherwise contribute less to the total loss).

Why this form: Flow matching with this specific interpolation xⁱ_{tⁱ} = tⁱ · xⁱ + (1 - tⁱ) · εⁱ defines a straight-line path (linear interpolation) between noise and data. The optimal vector field along this path is simply the constant direction (xⁱ - εⁱ) — the difference between the clean data and the noise. This makes the regression target simple and well-defined at every time step. Alternative formulations (e.g., standard diffusion with score matching) would predict the score ∇log p_t(x) rather than the velocity, requiring a different loss weighting. The flow matching formulation is chosen for compatibility with the Proteina baseline and because it supports both deterministic ODE sampling and stochastic SDE sampling (Equation 6) with the same trained network.

Architecture of v_θ. The flow decoder uses the same non-equivariant transformer architecture as Proteina, with 60M, 200M, or 400M parameters depending on the model scale (Table 6). Key dimensions: sequence representation dimension 512/768/1024, conditioning dimension 128/128/512, 12/12/16 attention heads, 12/15/18 transformer layers for 60M/200M/400M variants respectively. The conditioning embedding zⁱ is injected via adaptive layer normalization (AdaLN), following Peebles and Xie (2023): the conditioning vector is passed through a small MLP to predict scale and shift parameters that modulate the layer norm operations in each transformer block. This allows the conditioning to influence the decoder's internal representations at every layer.

Self-conditioning. In addition to zⁱ, the decoder can be conditioned on its own previous prediction (self-conditioning, Chen et al., 2022). During training, with 50% probability, the model's predicted clean data from the previous denoising step is provided as additional input. During sampling, the prediction from the previous time step x̂(x_t) = x_t + (1 - t)v_θ(x_t, t, z) is fed back as context. This provides a coarse estimate of the final structure that helps guide the denoising trajectory. The paper's ablation (Figure 8) shows self-conditioning consistently improves sc-RMSD across all protein lengths, and it is compatible with the multi-scale framework — it operates within each scale's flow matching process independently.

Training configuration. The model is trained with the Adam optimizer, learning rate 1e-4, no warmup, for 200K steps with batch size 15 per GPU on 8 H100 GPUs. The training data is the AFDB representative dataset (0.6M structures, lengths 32–256, pLDDT > 80), with an optional second-stage fine-tuning on a PDB designable subset (21K samples) for 5K steps. Pair representations and triangle-based modules from AlphaFold (Jumper et al., 2021) are disabled for computational efficiency and architectural alignment with Proteina.

Inference: Multi-Scale Structure Generation

At inference time, PAR generates a protein backbone through n autoregressive steps, each involving a flow matching sampling process:

Step-by-step generation:

Scale 1: The AR transformer takes only the learned bos token as input and produces z¹. The flow decoder initializes noise x¹_T ∼ N(0, I) ∈ ℝ^(size(1)×3) and runs the SDE or ODE sampler to progressively denoise it into x¹, conditioned on z¹. This produces the coarsest structural representation.
Scale 2: x¹ is upsampled to size(2) via interpolation. The AR transformer takes [bos, Up(x¹, size(2))] and produces z². The flow decoder samples x² from noise conditioned on z².
Scales 3 through n: Each scale i upsampled the previous scale's output, feeds all prior scales into the AR transformer to get zⁱ, and runs the flow decoder to generate xⁱ.
Output: After n steps, xⁿ is the full-resolution backbone x ∈ ℝ^(L×3).

KV caching. The AR transformer's key-value cache is maintained across scales for efficiency — the transformer does not need to recompute attention for earlier scales when processing later ones, since the input sequence simply grows by appending the new upsampled scale.

Sampling dynamics (SDE/ODE). The trained flow decoder v_θ can be used with either deterministic (ODE) or stochastic (SDE) sampling:

ODE sampling: dx_t = v_θ(x_t, t) dt. This follows the learned vector field deterministically from noise to data. It is fast but can produce lower-quality samples because there is no mechanism to correct errors that accumulate along the trajectory.
SDE sampling: The paper uses a stochastic differential equation that adds controlled noise during the denoising process:

$dx_t = v_θ(x_t, t) dt + g(t) s_θ(x_t, t) dt + \sqrt{2g(t)γ} dW_t$

where g(t) is a time-dependent scaling function, s_θ(x_t, t) is the score function (gradient of the log-probability), γ is a noise scaling parameter, and W_t is a standard Wiener process.

What it computes: The SDE has three terms: (1) the deterministic flow v_θ(x_t, t) dt that moves the sample toward the data distribution; (2) a score-based drift term g(t) s_θ(x_t, t) dt that pushes the sample toward high-probability regions; and (3) a noise injection term √(2g(t)γ) dW_t that adds stochasticity. The score function s_θ can be computed from the learned velocity v_θ without training a separate network: s_θ(x_t, t) = (t · v_θ(x_t, t) - x_t) / (1 - t). This relationship comes from the linear interpolation path and the connection between score matching and flow matching (Albergo et al., 2023).

Why this form: The SDE provides a tunable knob (γ) that controls the tradeoff between sample quality and diversity. When γ = 1, the SDE produces marginal distributions that match the ODE; when γ < 1, reduced stochasticity empirically improves designability (the fraction of generated structures that can be successfully refolded) at the cost of some diversity. The paper uses γ = 0.30 by default for reported results. Lowering γ from 0.45 to 0.30 in Table 1 improves designability from 88.0% to 96.0%. The SDE formulation is not novel to PAR — it is standard in flow matching literature — but PAR's contribution is demonstrating that different sampling strategies can be deployed at different scales.

Scale-dependent sampling orchestration (Table 2). The key efficiency insight: SDE sampling is necessary at the first (coarsest) scale to reliably establish the global topology, but ODE sampling suffices at finer scales because the coarse structure already places the model in a high-density region where deterministic refinement works well. The paper demonstrates this by comparing different sampling configurations for a 3-scale PAR generating length-150 proteins:

All SDE (400 steps per scale): 312 seconds, 97% designability — the quality baseline.
All ODE (400 steps per scale): 312 seconds, 28% designability — ODE alone fails because it cannot establish reliable global topology from scratch.
SDE at scale 1 (400 steps), ODE at scales 2–3 (2 steps each): 67 seconds, 97% designability — a 4.7× speedup over all-SDE with no quality degradation.
Single-scale baseline (Proteina, 400 SDE steps): 131 seconds, 97% designability — PAR achieves 2× speedup (67s vs 131s) at equivalent quality.

For length-200 proteins, PAR achieves a 2.5× speedup (68s vs 170s). The speedup increases with protein length because the computational savings at fine scales grow quadratically with sequence length (transformer attention cost), while the coarse scale cost remains fixed (always 64 points).

Why this works: The coarse SDE step explores the space of possible topologies stochastically, settling on a high-probability global layout. Once this layout is established, the fine-scale refinement is largely deterministic — interpolating the coarse layout provides strong spatial constraints, and ODE sampling efficiently resolves local detail without needing to explore alternative global configurations. The paper confirms this interpretation: reducing SDE steps at the first scale below 400 dramatically degrades designability (Figure 7, red curve drops to ~22% at 50 steps), while reducing steps at later scales has minimal impact when using ODE (blue and green curves remain above 95%).

Mitigating Exposure Bias

The paper identifies exposure bias — the mismatch between training on ground-truth context and inference on model-generated context — as a critical barrier to making multi-scale AR generation work for proteins. The problem is especially severe because structural errors compound across scales: a small inaccuracy in the coarse structure x¹ cascades into large-scale violations at x³. The authors' preliminary experiments showed that naive teacher forcing "greatly reduces the designability." To address this, PAR adapts two techniques from language and image AR modeling:

Noisy Context Learning (NCL)

During training, the ground-truth prior-scale inputs are corrupted with Gaussian noise before being fed to the AR transformer:

$xⁱ_{ncl} = wⁱ_{ncl} · xⁱ + (1 - wⁱ_{ncl}) · εⁱ_{ncl}$

where wⁱ_{ncl} ∈ [0, 1] is a randomly sampled noise weight for scale i and εⁱ_{ncl} ∼ N(0, I) is independent Gaussian noise. The corrupted structures replace the clean ones in the AR transformer input: zⁱ = T_θ([bos, Up(x¹_{ncl}, size(2)), ..., Up(xⁱ⁻¹_{ncl}, size(i))]).

What it computes: At each training step, every prior-scale structure is randomly corrupted — the weight wⁱ_{ncl} controls the mixing ratio; a weight of 1.0 means no corruption, 0.0 means pure noise. The AR transformer must learn to produce useful conditioning embeddings zⁱ even when the input context is degraded. This prevents the transformer from relying on perfectly accurate coarse structures that it won't have at inference time.

Why this form: Standard teacher forcing creates a "clean context" dependency — the model learns that if it sees a perfect coarse structure, it should generate a specific fine structure. But at inference, the coarse structure is model-generated and contains errors. Noisy context learning forces the model to be robust to input corruption; the wⁱ_{ncl} weights are sampled randomly per scale per training step, so the model experiences a wide range of corruption levels and learns to extract useful structural guidance even from noisy inputs. This is analogous to how denoising autoencoders learn robust representations by training on corrupted inputs.

Scheduled Sampling (SS)

In addition to NCL, scheduled sampling replaces ground-truth context with the model's own predictions during training:

$xⁱ_{pred} = xⁱ_t + (1 - tⁱ) · v_θ(xⁱ_t, tⁱ, zⁱ)$

With probability 0.5, the ground-truth xⁱ in the context for later scales (i+1, i+2, ...) is replaced with this predicted xⁱ_{pred}.

What it computes: When processing scale i during training, the flow decoder produces a one-step prediction xⁱ_{pred} of the clean data from the noised sample xⁱ_t. With 50% probability, this prediction replaces the ground-truth xⁱ in the input context for all subsequent scales. This means that at later scales, the AR transformer sometimes sees "real" prior structures and sometimes sees model-generated (and therefore imperfect) prior structures.

Why this form: This directly simulates the inference-time condition during training. At inference, all prior-scale contexts X^{<i} are model-generated. Scheduled sampling exposes the AR transformer to its own predictions during training, reducing the gap between training and inference distributions. The 50% probability balances exposure to both clean (ground-truth) and predicted (imperfect) contexts, preventing the model from either overfitting to clean context (teacher forcing) or training exclusively on poor-quality predictions (which could destabilize learning).

Interaction with NCL. The two techniques can be combined: when scheduled sampling selects a predicted context, that predicted context can additionally be corrupted with NCL noise. This provides a spectrum of context quality — from clean ground truth, to noisy ground truth, to clean predictions, to noisy predictions — giving the model broad experience with the types of imperfect context it will encounter at inference.

Empirical impact (Table 3). Evaluated on a 60M PAR trained for 100K steps:

Teacher forcing alone: sc-RMSD 2.20, FPSD vs. AFDB 37.64.
Adding NCL: sc-RMSD improves to 1.58, FPSD vs. AFDB drops to 23.69 (lower is better).
Adding NCL + SS: sc-RMSD further improves to 1.48, FPSD vs. AFDB at 24.59.

The substantial improvement in sc-RMSD (from 2.20 to 1.48) confirms that exposure bias is a first-order problem for structural generation — without these mitigations, the multi-scale AR approach would produce structures of significantly lower quality than the single-scale baselines it is compared against.

Implementation Details

Scale embedding (Section 6.1). Because a single shared flow decoder v_θ handles generation at all n scales, the model needs to know which scale it is operating at. A learnable embedding vector is assigned to each scale and concatenated with the conditioning zⁱ before injection. This embedding is not essential for the AR transformer (which sees the scale structure implicitly through the input sequence length), but is crucial for the decoder to modulate its behavior — at coarse scales the decoder should produce smooth, globally-consistent structures; at fine scales it should produce sharp, locally-detailed structures.

Interpolated position encoding (Section 6.1). Standard position encodings assume integer position indices 1, 2, ..., L. But PAR's downsampled structures have non-integer "positions" — a point at index 32.5 in a 64-point downsampling corresponds to roughly halfway between residues 32 and 33 in the original 256-residue protein. The interpolated position encoding computes sinusoidal embeddings at these fractional indices by interpolating between the embeddings of neighboring integer positions. This preserves the relative spatial relationships: a coarse-scale point at position 64.0 (the midpoint of a 128-residue protein) gets a position encoding close to what the actual residue 64 would get in the full structure, providing the model with location-awareness at all scales.

Two-stage training. Following Geffner et al. (2025), PAR uses a two-stage training procedure: (1) 200K steps on the AFDB representative dataset (0.6M structures) for broad structural diversity, and (2) 5K steps of fine-tuning on a PDB designable subset (21K structures) for high designability. The PDB subset is curated by filtering for structures that are actually designable (can be recovered by ProteinMPNN + ESMFold) — this fine-tuning stage biases the model toward producing structures that can be realized with actual amino acid sequences, which is important for practical protein design applications.

Fair comparison with baselines. For the unconditional generation benchmark, the paper trains Proteina (the primary single-scale flow baseline) from scratch under identical conditions (same architecture without pair representations/triangle updates, same compute, same data) and achieves similar or better performance than reported in the original paper. All other baseline results (FrameDiff, RFDiffusion, ESM3, Genie2) are taken directly from Geffner et al. (2025) without re-training. This ensures that any performance differences between PAR and Proteina reflect the multi-scale autoregressive framework rather than training data, compute, or architectural differences.

4. Key Insights and Innovations

Innovation 1: Autoregressive Generation Can Work for Protein Backbones — If You Factorize Across Scales, Not Residues

The dominant assumption in the protein structure generation field — reinforced by the near-total absence of AR methods in the literature — has been that autoregressive modeling is fundamentally mismatched with protein biochemistry. The reasoning is straightforward: AR models impose a unidirectional dependency (each element depends only on prior elements), but protein residues exhibit strong bidirectional spatial dependencies (residues far apart in sequence are often adjacent in 3D and mutually constrain each other's positions). The one prior AR attempt (Gaujac et al., 2024) tacitly accepted this constraint, applying a standard causal transformer to discretized structure tokens generated left-to-right — and the results, while novel, did not displace diffusion models as the dominant paradigm.

PAR's core intellectual move is to change the unit of autoregression from "residue" to "scale." Rather than generating one residue at a time in sequence order, PAR generates an entire structural representation at a given resolution (all residues at that scale, simultaneously via flow matching) conditioned on all previously generated coarser representations. This reframing dissolves both objections to AR modeling in one stroke. The bidirectional dependency problem is resolved because within each scale, the flow decoder sees all positions simultaneously through its transformer attention — residues 10 and 150 mutually constrain each other within the same scale, just as they would in a diffusion model. The unidirectional constraint applies only across scales (coarse → fine), which is physically sensible: the global fold topology should constrain local detail, but not vice versa. The discretization problem is resolved because the flow decoder operates directly on continuous Cα coordinates — no tokenizer, no quantization, no information loss.

This is not an incremental improvement on Gaujac et al. (2024). It is a fundamentally different decomposition of the generation problem — from token-level autoregression to scale-level autoregression — that changes what the AR model actually predicts. The flow decoder at each scale is essentially a single-scale flow matching model; what makes the system autoregressive is the conditioning chain that passes structural information from coarser to finer scales. Section 5 makes this explicit: "PAR provides a more general framework, incorporating flow-based models as a special case when restricted to a single scale." If you set n = 1 (only the full-resolution scale), the AR transformer produces a single conditioning vector and the flow decoder is identical to Proteina. The multi-scale extension adds autoregressive structure without changing the underlying generative machinery — the flow decoder architecture, the flow matching loss, the SDE/ODE sampling, and even self-conditioning all remain unchanged.

The significance of this reframing goes beyond PAR itself. It opens a design space that the field had not previously explored: any generative model that operates at a single scale can, in principle, be embedded in a multi-scale autoregressive framework by training it to generate structures at multiple resolutions conditioned on coarser representations. This is not specific to flow matching — diffusion models, score-based models, or even GANs could be plugged in as the per-scale decoder. The AR transformer is the new component, but its role (producing conditioning embeddings from structural history) is architecturally simple and the paper shows it does not need to be large (60M parameters suffices; scaling to 400M does not help, per Table 12). The framework is modular: improvements to the flow decoder (better architectures, better sampling) and improvements to the AR transformer (better context encoding, better exposure bias mitigation) can be pursued independently.

Empirically, the unconditional generation results in Table 1 confirm that this reframing does not sacrifice quality relative to pure flow models — PAR achieves competitive or better FPSD (161.0 vs. PDB after fine-tuning, compared to 271.3 for Proteina at 400M) and comparable designability. But the more distinctive evidence for the value of the scale-level factorization comes from the behaviors that single-scale models cannot replicate: the interpretable coarse-to-fine generation visualized in Figure 2, the zero-shot prompt and motif tasks (Figures 3 and 4), and the scale-dependent SDE/ODE orchestration that enables 2.5× faster sampling (Table 2). These capabilities are direct consequences of the architectural decision to factorize across scales rather than residues.

Innovation 2: Multi-Scale Generation Creates an Interpretable "Sculpting" Process That Enables Zero-Shot Conditional Generation

Single-scale diffusion and flow models treat protein backbone generation as a monolithic denoising process: start from pure noise in the full-dimensional space and progressively refine all residues simultaneously. There is no explicit notion of "coarse structure" versus "fine detail" — the denoising trajectory may implicitly discover coarse-to-fine structure (diffusion models are known to generate low frequencies before high frequencies), but the model architecture does not enforce or expose this hierarchy. Consequently, conditioning such models on partial structural specifications (a motif, a coarse layout, a set of spatial constraints) requires integrating those specifications into the denoising process, which typically means training with those specific conditioning signals. A model trained for unconditional generation cannot, at test time, accept a 16-point coarse layout and generate a compatible full backbone — it was never trained to map from partial to complete structures.

PAR's multi-scale autoregressive structure inverts this limitation. Because the model is explicitly trained to generate coarse scales before fine scales, and because the AR transformer learns to map from coarser structural context to conditioning for the next finer scale, the model naturally accepts partial specifications at any scale and completes them autoregressively. If you provide x¹ (a 16-point coarse layout) at the first scale, the AR transformer can compute z² from this provided context, and the flow decoder can generate x², x³, and so on — exactly as it does during unconditional generation, except the first scale is given rather than generated. No fine-tuning is needed because the model was trained on exactly this task: predicting scale i given scales 1 through i-1.

This is a fundamentally different relationship between pretraining and downstream tasks than what exists in diffusion-based protein models. In diffusion models, unconditional pretraining teaches the model to denoise structures, but conditional generation requires learning a separate mapping — typically by training with conditioning masks or by learning to infill missing regions. Each conditioning type (motif scaffolding, length control, symmetry constraints) requires its own training procedure. PAR's multi-scale AR framework collapses this distinction: any partial structural specification that can be expressed at some scale is a valid prefix of the autoregressive generation process. The "task" is just starting the autoregressive chain from a provided prefix rather than from bos.

The paper demonstrates this with two qualitatively different conditioning types. Prompt-based generation (Figure 3, Section 4.2): providing as few as 16 points (a extremely coarse layout specifying only the global fold topology) as the first-scale structure causes PAR to generate complete backbones that respect the specified coarse arrangement while filling in novel local detail. Table 5 quantifies this structural consistency, showing that the AR transformer achieves better prompt-following (higher TM-score vs. prompt) than directly feeding the prompt to the flow decoder — the autoregressive encoding across scales produces coherent structural guidance. Motif scaffolding (Figure 4, Section 4.2): providing atomic coordinates for a functional motif (e.g., a binding pocket) as partial context at each scale, PAR generates diverse scaffolds that preserve the motif with high fidelity while introducing novel structural elements elsewhere. Table 10 benchmarks this zero-shot against methods that require fine-tuning (Proteina, Genie2, RFDiffusion, FrameFlow) — PAR solves 13 out of 24 benchmark problems at ≥1% success rate in the zero-shot setting, compared to 11 problems for the best fine-tuned baseline, despite generating only 100 samples vs. 1000 for the baselines.

The intellectual significance here is not that conditional generation is possible (diffusion models do it) but that the multi-scale AR framework makes conditional generation a natural consequence of the architecture rather than an additional training objective. This aligns PAR with the "zero-shot generalization" narrative that has driven LLM adoption — the idea that a single pretrained model can perform diverse tasks by conditioning on different input contexts, without task-specific training. For protein design, where novel conditioning signals arise frequently (a new binding target, a new structural constraint, a new symmetry requirement), this flexibility is practically valuable. It means researchers can specify constraints at the level of granularity that is natural for their problem (coarse topology for global architecture design, atomic coordinates for active site engineering) and the model adapts without retraining.

Innovation 3: Exposure Bias Is a First-Order Barrier for Multi-Scale AR in Structured Domains — and Requires Deliberate Mitigation

Exposure bias (the train-test mismatch from teacher forcing) is a known problem in autoregressive sequence generation, well-documented in NLP (Arora et al., 2022) and addressed in image generation (Ren et al., 2025) with techniques like scheduled sampling. The standard narrative is that exposure bias causes a gradual degradation in output quality as errors accumulate across autoregressive steps — problematic, but often manageable with simple interventions or large-scale training.

PAR's experience reveals that in structured, continuous domains like protein backbone generation, exposure bias is qualitatively more severe. The paper's preliminary experiments showed that naive teacher forcing "greatly reduces the designability of generated structures" (Section 3.3). Table 3 quantifies this: a 60M PAR trained with teacher forcing alone achieves an sc-RMSD of 2.20 and FPSD vs. AFDB of 37.64, while adding noisy context learning (NCL) alone improves sc-RMSD to 1.58 and FPSD to 23.69. Adding scheduled sampling (SS) on top of NCL further improves sc-RMSD to 1.48.

The magnitude of the degradation (sc-RMSD worsening by ~50% from 1.48 to 2.20) is larger than what is typically reported in language modeling, and the paper's analysis in Section 8.7 provides a mechanistic explanation for why. Three factors compound: (1) Geometric error propagation: an error in the coarse-scale structure x¹ affects all subsequent scales — a small misplacement of a helix at 64-point resolution cascades into large RMSD deviations at 256-point resolution because the fine-scale decoder is conditioned on the incorrect coarse layout. This is fundamentally different from language, where a wrong word early in a sentence may or may not affect later words depending on semantic dependencies. (2) Limited training data: the AFDB dataset contains ~0.6M structures, which is orders of magnitude less than typical image datasets (ImageNet has 1.28M images with spatial dimensions; protein structures have much higher effective dimensionality). With less data, the AR transformer more easily overfits to the clean ground-truth context it sees during teacher forcing, making it brittle when fed imperfect predictions at inference. (3) High precision requirements: protein designability requires sub-2Å RMSD accuracy — small coordinate errors that would be imperceptible in image generation (where VAE latents smooth over imprecisions) can render a protein non-designable. The paper explicitly notes (Section 8.7) that increasing the AR transformer from 60M to 400M worsens performance because the larger model overfits the clean context more aggressively, intensifying exposure bias.

This finding has implications beyond PAR. It suggests that any multi-scale autoregressive approach to structured, continuous data — 3D molecular generation, mechanical part design, architectural layout — will face exposure bias as a first-order barrier rather than a minor inconvenience. The geometric compounding of errors across scales is a general phenomenon whenever coarse-scale predictions constrain fine-scale generation. The paper's solution (NCL + SS) is adapted from prior work but the diagnostic insight — that exposure bias severity scales with structural precision requirements and data limitations — is a conceptual contribution that should inform future work in these domains.

The paper also provides an interesting negative result that sharpens this insight: the ReST^EM experiment in Appendix K (not fully detailed in the main text but referenced) shows that RL-based fine-tuning of the revision model degrades performance with sequential revisions, likely because on-policy data collection amplifies spurious correlations. This reinforces the message that exposure bias mitigation is delicate and that standard techniques from less-structured domains may not transfer directly.

Single-scale generative models apply a uniform computational budget to all parts of the generation process — typically 400–1000 denoising steps, regardless of whether the model is establishing global structure or refining local details. This is computationally wasteful because these two phases have fundamentally different requirements: establishing a reliable global topology benefits from stochastic exploration (SDE sampling) to avoid getting trapped in poor local optima, while refining local detail given a fixed topology is largely deterministic and can be done efficiently with ODE sampling.

PAR's multi-scale architecture makes this decomposition explicit and exploitable. Because the coarsest scale (size(1) = 64) captures the global fold topology and operates on a short, fixed-length representation, the model can invest heavily in stochastic SDE sampling at this scale (400 steps) to reliably explore the space of possible topologies. Once this topology is established, the finer scales (size(2) = 128, size(3) = L) need only refine the layout — a task that ODE sampling handles efficiently, requiring as few as 2 steps per scale without quality degradation. Table 2 demonstrates this: SDE at scale 1 + ODE at scales 2–3 (400/2/2 steps) achieves 97% designability at length 150 in 67 seconds, compared to 312 seconds for all-SDE and 131 seconds for the single-scale Proteina baseline.

This is more than an engineering optimization — it is a conceptual reframing of what different scales of a hierarchical generation process need computationally. The paper's interpretation (Section 4.3 and Figure 7) is that PAR effectively decomposes backbone generation into "coarse topology formation" (scale 1, where SDE is necessary) and "efficient structure refinement" (scales 2+, where ODE suffices). The evidence supports this interpretation: ODE-only sampling across all scales yields only 28% designability (Table 2), confirming that stochastic exploration at the coarse scale is essential. Reducing SDE steps at the first scale below 400 degrades designability sharply (Figure 7, red curve drops to ~22% at 50 steps), confirming that topology formation requires substantial stochastic computation. But once the topology is set, the refinement at finer scales is robust to drastic step reduction — 2 ODE steps at scales 2–3 maintain 97% designability.

This finding implies a more general principle that the paper does not fully explore but that follows naturally: in hierarchical generative models, different levels of the hierarchy have different computational needs, and allocating a uniform budget across levels is suboptimal. This is analogous to how some vision architectures allocate more computation to early layers (where features are built) than later layers (where they are refined), but applied to the generation process rather than the model architecture. The paper's specific demonstration — SDE for topology, ODE for refinement — is likely domain-specific (protein structures have a natural coarse-to-fine hierarchy that maps well to this decomposition), but the principle of scale-dependent compute allocation should generalize to other hierarchical generation tasks.

The practical impact is significant: PAR's 2.5× sampling speedup over single-scale models (and up to 4.7× when comparing to all-SDE within PAR) makes the multi-scale approach not just conceptually appealing but computationally advantageous. The speedup increases with protein length because the coarse scale remains fixed at 64 points while the fine-scale savings grow quadratically — for very long proteins (500+ residues), the advantage would be even larger.

Innovation 5: The Autoregressive Transformer Serves as a Learned "Structural Router" That Integrates Information Across Scales

A subtle but important finding emerges from the attention map analysis (Figure 6) and the ablation comparing AR transformer encoding to direct input (Table 5). The AR transformer is not simply a static encoder of prior-scale structures — it learns a scale-attention pattern that dynamically routes structural information from coarser to finer scales during generation.

Figure 6 reveals a consistent and interpretable attention structure: (1) most scales barely attend to the first scale (bos token), which carries little structural signal; (2) each scale primarily attends to the immediately previous scale, which contains the richest and most recent structural context; (3) despite this recency bias, non-negligible attention is retained to earlier scales, enabling the model to maintain global structural consistency. This pattern is not hard-coded — it emerges from training and persists across protein lengths (also visible in the shorter-protein attention maps in Figure 10). It suggests that the AR transformer learns to treat the most recent coarser scale as the primary structural guide while using earlier scales as long-range consistency constraints.

The ablation in Table 5 confirms that this learned routing matters for structural fidelity. When prior-scale structures are fed directly to the flow decoder (bypassing the AR transformer), the structural consistency with the prompt degrades: RMSD vs. reference increases from 1.45 to 2.13 (32–64 length range) and from 5.75 to 6.51 (128–256 range), while TM-score vs. prompt decreases from 0.61 to 0.59 (128–256 range). The AR transformer is doing more than just passing structural information through — it is actively integrating across scales to produce conditioning that improves coherence.

This finding positions the AR transformer as a learned structural router rather than a mere encoder. Its role is not to extract features from individual scales (which could be done with a simpler encoder) but to decide how much information from each prior scale is relevant for the current generation step, and to fuse that information into a conditioning vector that optimally guides the flow decoder. This is a more nuanced role than the MAR architecture it is based on (Li et al., 2024), where the AR model produces conditioning for a single-scale diffusion process — here, the AR transformer must handle multiple input scales, each at different resolutions, and produce conditioning that coherently guides multi-scale refinement.

The practical implication is that the AR transformer does not need to be large — 60M parameters suffices, and scaling to 400M actually degrades performance (Table 12) due to increased overfitting to clean context. This is consistent with its role as a context integrator rather than a primary generator: the heavy lifting of coordinate generation is done by the flow decoder, which does benefit from scaling (Figure 5 shows monotonic improvement in FPSD and sc-RMSD as the decoder grows from 60M to 400M). The architectural separation of concerns — small context router, large structure generator — is a design principle that emerges from the empirical results rather than being imposed a priori, and it suggests an efficient allocation of parameters for future multi-scale generative models in other domains.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the AFDB representative dataset as processed by Proteina (Geffner et al., 2025), containing approximately 0.6M protein structures with pLDDT > 80 and sequence lengths ranging from 32 to 256 residues. The data is split 98:1:1 for training, validation, and testing. For PDB fine-tuning, the authors reproduce Proteina's filtering protocol to curate a designable subset of 21K structures from the PDB, since the original Proteina PDB dataset is not publicly available. For long-protein experiments, an additional PDB-derived dataset of 26K structures with lengths 256–768 residues is curated following the same filtering procedure.
Base model(s). PAR uses two trainable components: an autoregressive transformer (AR transformer) of 60M parameters for scale-wise conditioning, and a flow-based backbone decoder (v_θ) in three size variants — 60M, 200M, and 400M parameters — all using the non-equivariant transformer architecture from Proteina. The 400M decoder configuration is the default for main results. The AR transformer remains fixed at 60M across all variants because scaling it further does not improve performance (Section 8.7, Table 12). The primary single-scale baseline, Proteina, uses the same architecture and training setup to ensure fair comparison.
Metrics. Seven metrics are reported, following the evaluation protocol from Geffner et al. (2025):
- Designability (%): For each generated structure, 8 sequences are designed with ProteinMPNN (temperature 0.1) and folded with ESMFold. A structure is designable if the minimum RMSD between any predicted structure and the original is < 2 Å. Reported as both the percentage of designable structures and the average minimum RMSD (sc-RMSD) across all samples.
- FPSD (Fréchet Protein Structure Distance): Analogous to FID in image generation, this measures the Wasserstein distance between generated and reference structure distributions in a fold-classifier feature space. Lower is better. Reported against both PDB and AFDB reference sets.
- fS (Protein Fold Score): Inspired by Inception Score, this captures both sample quality (confident fold predictions) and diversity (entropy across fold classes). Higher is better. Reported as three components: C (secondary structure content), A (accuracy), T (total score).
- Diversity: Average pairwise TM-score among all designable structures at each length, then averaged across lengths. Lower TM-score indicates higher diversity.
- Secondary Structure (%): Proportions of α-helices and β-sheets computed using P-SEA via Biotite, reported as (α/β) percentages.
- TM-score vs. Prompt: For prompted generation, structural similarity between the input prompt layout and the generated structure.
- RMSD vs. Reference: For prompted generation, RMSD between generated structures and reference structures when using real downsampled proteins as prompts.
For distributional metrics (FPSD, fS), 5,000 structures are sampled without designability filtering (125 per sequence length from 60 to 255, step size 5). For other metrics, 500 structures are generated (100 each at lengths 50, 100, 150, 200, 250).
Baselines. The paper compares against five baseline methods spanning three categories (Table 1):
- Frame-based diffusion: FrameDiff (Yim et al., 2023b, 17M parameters), RFDiffusion (Watson et al., 2023, 60M parameters).
- Multimodal protein language models: ESM3 (Hayes et al., 2025, 1.4B parameters), which uses discrete structure tokens.
- Diffusion/flow-based Cα generators: Genie2 (Lin et al., 2024, 16M parameters), Proteina (Geffner et al., 2025, 200M and 400M parameters). Proteina is the most direct comparison as it shares architecture, training data, and evaluation protocol with PAR. All baseline results except Proteina are taken directly from Geffner et al. (2025); Proteina is trained from scratch under identical conditions for fair comparison.
Generation budget / compute accounting. The paper does not use a unified compute budget in the vein of FLOPs-matched comparisons. Instead, comparisons use matched model sizes (200M and 400M parameter Proteina vs. PAR with equivalent decoder sizes) and matched training regimes (200K steps on AFDB, 5K steps on PDB). Sampling efficiency is compared by wall-clock time (seconds) to generate 100 samples at fixed lengths, not by FLOP counts. For the scaling analysis, training budget is measured in training steps (200K, 400K, 600K) and model parameters (60M, 200M, 400M decoder parameters).
Cross-validation / statistical protocol. No cross-validation is employed for the unconditional generation benchmark. Results are reported from a single training run per configuration, with evaluation on a fixed test set across 500 generated structures. The paper does not report error bars or confidence intervals for any metrics. For the motif scaffolding benchmark (Table 10), 100 samples are generated per problem (compared to 1000 for baselines) and reported as success rates — but no statistical test is applied to assess whether differences are significant.

Main Quantitative Results

Unconditional Backbone Generation (Table 1)

The headline result is that PAR achieves competitive or superior distributional fidelity compared to existing methods. The 400M PAR model fine-tuned on PDB (PAR_pdb) achieves an FPSD of 161.0 against the PDB reference set, substantially better than the next-best method Proteina at 400M (271.3 vs. PDB, 272.6 vs. AFDB), and designability of 96.6% with sc-RMSD of 1.04 Å. Before PDB fine-tuning, PAR achieves 96.0% designability (sc-RMSD 1.01) and FPSD of 313.9 vs. PDB and 296.4 vs. AFDB.

Comparing PAR to Proteina at matched model sizes (both 400M, both without PDB fine-tuning): PAR achieves 96.0% designability vs. Proteina's 92.6%, with similar sc-RMSD (1.01 vs. 1.09). On FPSD, Proteina performs better on the PDB reference (271.3 vs. 313.9) but PAR performs better on the AFDB reference (296.4 vs. 272.6), indicating that the two models capture somewhat different aspects of the structural distribution. After PDB fine-tuning, PAR's FPSD advantage over Proteina becomes unambiguous (161.0 vs. 271.3 for PAR_pdb vs. Proteina 400M; note however that Proteina's PDB-fine-tuned results are not reported in Table 1, so this comparison is between PAR_pdb and the AFDB-trained Proteina).

On secondary structure composition, PAR generates 66.3% α-helices and 8.9% β-sheets (400M, pre-fine-tune), compared to 65.1%/9.5% for Proteina 400M. After PDB fine-tuning, this shifts to 50.2%/16.7% for PAR_pdb, reflecting the different secondary structure distribution in the PDB subset. PAR's diversity (TM-score 0.39) is comparable to Proteina's (0.37).

Notably, adjusting the noise scaling parameter γ in the SDE sampler from 0.45 to 0.30 improves PAR's designability from 88.0% to 96.0% (400M, pre-fine-tune), though this comes with a tradeoff in distributional metrics — FPSD worsens from 231.5 to 313.9 vs. PDB. The paper uses γ = 0.30 as the default for reported designability, but reports both FPSD values in Table 1 (the γ = 0.45 row shows the better distributional fidelity).

The interpretability of PAR's generation process is shown in Figure 2 for a 5-scale model (S = {L/16, L/8, L/4, L/2, L}) generating structures at lengths 50, 100, 200, and 250. The coarse-to-fine progression is visually apparent, with early scales establishing global topology and later scales adding secondary structure elements and local detail.

Scaling Behavior (Figure 5)

The paper demonstrates that PAR exhibits favorable scaling with both model size and training duration, evaluated on FPSD (vs. PDB and AFDB), fS(T), and sc-RMSD. Three decoder sizes (60M, 200M, 400M parameters) and three training durations (200K, 400K, 600K steps) are tested with a fixed 3-scale configuration S = {64, 128, 256}.

For FPSD vs. PDB (Figure 5a), performance improves monotonically with both axes: at 200K steps, FPSD drops from ~300 (60M) to ~255 (200M) to ~230 (400M); at 600K steps and 400M parameters, FPSD reaches approximately 187. For FPSD vs. AFDB (Figure 5b), the same trend holds, reaching approximately 170 at 600K steps and 400M parameters. The fS(T) score (Figure 5c) increases with larger models and more training, reflecting improving fold diversity and quality. sc-RMSD (Figure 5d) shows that extending training duration alone offers negligible gains — the 60M model's sc-RMSD stays flat around 1.5 across all training durations — whereas increasing model size substantially reduces sc-RMSD (reaching ~1.0 at 400M).

A practical finding from these scaling experiments: the paper reports that "scaling the autoregressive transformer has minimal impacts on the evaluation results" (Section 4.3). This allows prioritizing decoder capacity over AR transformer capacity, since the decoder's size drives quality improvements while the AR transformer's size (fixed at 60M) does not. Section 8.7 (Table 12) confirms that with a 400M decoder, increasing the AR transformer from 60M to 400M does not improve designability (96.0% vs. 87.8% with 400M AR and 60M decoder — the performance drops when AR is scaled but decoder is not).

Sampling Efficiency with SDE/ODE Orchestration (Table 2)

The paper quantifies how PAR's multi-scale structure enables reducing sampling steps at fine scales without quality loss. For 3-scale PAR on length-150 proteins:

All SDE (400/400/400): 312 seconds, 97% designability.
All ODE (400/400/400): 312 seconds, 28% designability — ODE alone cannot establish reliable topology.
S/O/O (400/2/2): SDE at scale 1 (400 steps), ODE at scales 2–3 (2 steps each): 67 seconds, 97% designability — a 4.7× speedup over all-SDE internally, and a 1.96× speedup over the single-scale Proteina baseline (131 seconds, 97% designability with 400 SDE steps).
S/S/O (400/400/2): 184 seconds, 99% designability — using SDE at scales 1–2 and ODE at scale 3 preserves quality with moderate speedup.

On length-200 proteins, the S/O/O configuration (400/2/2) achieves 94% designability in 68 seconds vs. Proteina's 92% designability in 170 seconds — a 2.5× speedup. A more conservative S/S/O configuration (400/400/2) achieves 91% designability in 186 seconds.

The configuration SDE 400/400/2 (reducing only the last scale to 2 ODE steps but keeping SDE at scales 1–2) yields 0% designability, indicating that abruptly switching from 400 SDE steps to 2 ODE steps at the final scale is harmful when the prior scale was also stochastic — the structural context from SDE at scale 2 may be too noisy for ODE at scale 3 to clean up. The successful configurations (S/O/O) place the SDE step at the coarsest scale only.

Figure 7 provides additional detail: naively reducing SDE steps at all scales degrades designability rapidly (dropping to ~22% at 50 steps per scale), ODE-only across all scales plateaus at low designability (~28%), while SDE at scale 1 + ODE at scales 2–3 maintains high designability (>95%) even with drastically reduced steps at fine scales — but only when the first scale retains 400 SDE steps. Reducing the first scale's SDE steps below 400 (even with ODE at later scales) causes designability to degrade, confirming that establishing the coarse topology requires substantial stochastic computation.

Zero-Shot Prompted Generation (Figure 3, Table 5)

The paper demonstrates two types of zero-shot conditional generation enabled by PAR's multi-scale architecture. For prompt-based generation (Figure 3), a 5-scale PAR (S = {16, 32, 64, 128, 256}) is given a 16-point coarse layout as the first-scale structure (either from downsampling a real protein or manually specified) and autoregressively generates the remaining scales. The generated structures preserve the global layout specified by the prompt while introducing novel local details. The paper notes that longer prompts (e.g., 32 points) can be used for finer-grained control.

Table 5 quantifies the structural consistency of prompted generation, comparing two strategies for encoding prior-scale context: direct input (prior-scale structures fed directly to the flow decoder) vs. transformer encoding (processing through the AR transformer). The AR transformer consistently achieves better structural fidelity: for structures in the (32, 64] length range, RMSD vs. reference is 1.45 with transformer encoding vs. 2.13 with direct input, and TM-score vs. prompt is 0.60 vs. 0.58. For the (128, 256] range, RMSD is 5.75 vs. 6.51 and TM-score is 0.61 vs. 0.59. The reference row (downsampling the same protein and regenerating) achieves TM-scores of 0.60, 0.61, and 0.59 across the three length bins — the AR-transformer-encoded model essentially matches this self-reconstruction consistency, suggesting that the 16-point prompt captures the essential structural identity.

Zero-Shot Motif Scaffolding (Figure 4, Table 10)

For motif scaffolding, PAR uses a 5-scale model and teacher-forces the ground-truth motif coordinates at each scale before propagating to the next scale. To avoid clashes, the ground-truth motif residues and generated motif segments are superimposed before replacement. Without any fine-tuning or conditioning beyond this teacher-forcing, PAR generates scaffolds that preserve the motif with high fidelity while introducing structurally diverse surrounding elements (Figure 4 shows examples where a preserved helix motif is scaffolded with new β-sheets and loops that differ from the original structure).

Table 10 benchmarks this zero-shot capability against methods that require fine-tuning for motif scaffolding (Proteina, Genie2, RFDiffusion, FrameFlow) on 24 problems from Watson et al. (2023). PAR solves 13 out of 24 problems at ≥1% success rate, compared to 11 for Proteina (the best fine-tuned baseline). On individual problems, PAR achieves notable success rates: 10% on 6EXZ_long, 8% on 6EXZ_med, 6% on 3IXT, 6% on 6E6R_short, 4% on 1YCR, 4% on 2KL8. However, PAR achieves 0% on 11 of the 24 problems, including 1PRW, 1BCF, 5TPN, 5IUS, and others. The paper notes that PAR was evaluated with only 100 samples per problem while baselines used 1000 samples, making the zero-shot results "conservative" — but this also means the comparison is not strictly apples-to-apples in terms of sample budget.

Beyond the success rates, the paper emphasizes a qualitative difference: PAR performs motif scaffolding in a zero-shot setting while all baselines require task-specific training or fine-tuning. This is the paper's primary claim about this experiment — not that PAR outperforms fine-tuned methods (it doesn't consistently), but that PAR achieves competitive performance without any task-specific adaptation, demonstrating the zero-shot generalization that the multi-scale AR framework enables.

Exposure Bias Mitigation (Table 3)

Table 3 ablates the strategies for mitigating exposure bias using a 60M PAR trained for 100K steps, evaluated with ODE sampling:

Teacher forcing alone: sc-RMSD 2.20, FPSD vs. PDB 99.66, FPSD vs. AFDB 37.64, fS-(C/A/T) 2.53/5.56/29.67.
Adding Noisy Context Learning (NCL): sc-RMSD improves to 1.58, FPSD vs. PDB drops to 89.70, FPSD vs. AFDB drops to 23.69, fS-(C/A/T) 2.54/5.85/28.37.
Adding both NCL and Scheduled Sampling (SS): sc-RMSD improves to 1.48, FPSD vs. PDB at 90.66, FPSD vs. AFDB at 24.59, fS-(C/A/T) 2.54/5.84/28.77.

The dominant effect comes from NCL (reducing sc-RMSD from 2.20 to 1.58), with SS providing a smaller additional improvement (1.58 → 1.48). FPSD improves substantially with NCL (from 37.64 to 23.69 vs. AFDB) but SS provides no additional FPSD benefit (24.59), suggesting that NCL primarily addresses the distributional quality while SS primarily helps with per-sample structural accuracy.

Multi-Scale Formulation Ablation (Table 4)

Table 4 ablates the scale configuration for a 60M PAR, comparing different numbers of scales and definition strategies:

2 scales (S = {64, 256}): designability 83.0%, sc-RMSD 1.38, FPSD vs. PDB 282.85.
3 scales (S = {64, 128, 256}): designability 85.0%, sc-RMSD 1.39, FPSD vs. PDB 279.63.
4 scales (S = {64, 128, 192, 256}): designability 77.8%, sc-RMSD 1.55, FPSD vs. PDB 296.70.
5 scales (S = {64, 96, 128, 192, 256}): designability 81.0%, sc-RMSD 1.51, FPSD vs. PDB 276.00.
Ratio-based 3 scales (S = {L/4, L/2, L}): designability 86.4%, sc-RMSD 1.49, FPSD vs. PDB 310.64.

The 3-scale, length-defined configuration (S = {64, 128, 256}) achieves the best balance of FPSD (279.63) and designability (85.0%). Increasing to 4 or 5 scales degrades designability (77.8% and 81.0% respectively), which the authors attribute to error accumulation across more autoregressive steps and increased exposure bias. The ratio-based definition achieves higher designability (86.4%) but substantially worse FPSD (310.64 vs. 279.63), indicating it produces individually designable structures that are less representative of the overall distribution. This supports the paper's choice of 3 scales defined by length as the default.

AR Transformer vs. Direct Input (Table 5)

The ablation in Table 5 (discussed above under prompted generation) demonstrates that the AR transformer actively improves structural consistency rather than merely passing through context. When generating structures from 16-point prompts (5-scale PAR), using the AR transformer to encode prior-scale context achieves RMSD vs. reference of 1.45/2.72/5.75 across three length bins, compared to 2.13/3.38/6.51 when prior-scale structures are fed directly to the flow decoder. The TM-score vs. prompt improves from 0.58/0.61/0.59 to 0.60/0.64/0.61. The improvement is particularly notable in the medium length range (64–128 residues), where TM-score jumps from 0.61 to 0.64 — the AR transformer is most helpful when there is substantial structure to be inferred from the coarse prompt at intermediate resolutions.

Attention Map Interpretation (Figure 6, Figure 10)

Figure 6 visualizes the average attention scores of the AR transformer across 5 scales for proteins with lengths in (128, 256], aggregated over 50 samples. The attention patterns reveal three systematic behaviors: (i) most scales barely attend to the first scale (the bos token input), since it carries minimal structural signal; (ii) each scale primarily attends to the immediately preceding scale (the strongest attention blobs are on the sub-diagonal), which contains the most recent and structurally informative context; (iii) despite this recency bias, non-negligible attention is retained to earlier scales (visible as off-diagonal signal), particularly from scale 3 to scale 1 and from scale 4 to scale 2. Figure 10 shows the same patterns for shorter proteins (lengths in (32, 64] and (64, 128]), confirming that the attention structure is consistent across protein lengths.

Ablation Studies and Robustness Checks

Self-conditioning (Figure 8): The paper ablates whether self-conditioning (feeding the decoder's own previous prediction as additional input) is compatible with and additive to the multi-scale framework. Figure 8 shows that adding self-conditioning to a 60M PAR consistently reduces sc-RMSD across all tested protein lengths (50, 100, 150, 200, 250), with the gap being roughly 0.1–0.3 Å across lengths. The improvement is present at all lengths and the trend lines are roughly parallel, indicating that self-conditioning and multi-scale autoregression address complementary aspects of generation quality. This confirms that PAR remains compatible with standard techniques from the flow matching literature.

Scale embedding (Table 11): Removing the learnable scale embedding from the 400M PDB-fine-tuned model reduces designability from 96.6% to 92.8% (sc-RMSD degrades from 1.04 to 1.16) and worsens FPSD vs. PDB from 161.0 to 175.1. However, removing the scale embedding enables "scale-agnostic" inference — the same model can be run with 5 scales instead of the 3 it was trained with. When doing so, FPSD remains relatively stable (177.0 vs. PDB) but designability drops substantially to 72.6% (sc-RMSD 1.74), indicating that while the model can approximately handle unseen scale configurations at the distribution level, the per-sample structural quality degrades significantly.

AR transformer vs. decoder size (Table 12, Section 8.7): The paper ablates the allocation of parameters between the AR transformer and the flow decoder:

400M AR transformer + 60M decoder: sc-RMSD 1.26, designability 87.8%.
60M AR transformer + 400M decoder: sc-RMSD 1.01, designability 96.0%.
60M AR transformer + 60M decoder: sc-RMSD 1.19, designability 92.6%.

Scaling the decoder drives improvements (60M → 400M decoder improves designability from 92.6% to 96.0%), while scaling the AR transformer alone (60M + 60M decoder vs. 400M AR + 60M decoder) degrades designability from 92.6% to 87.8%. The paper attributes this to exposure bias: larger AR transformers overfit more aggressively to the clean ground-truth context during teacher forcing, making them more brittle when fed imperfect predictions at inference. This finding motivates the design choice to fix the AR transformer at 60M across all model scales.

Sequence-based downsampling preserves spatial relationships (Table 13): The paper validates that 1D sequence-based downsampling adequately preserves 3D pairwise spatial relationships by comparing pairwise distance maps computed from downsampled coordinates vs. downsampled full-resolution distance maps. Across downsampling sizes 16, 32, 64, and 128, the LDDT between the two distance map representations is consistently 1.0 (perfect at the threshold used), and the RMSE values are small: 0.362, 0.275, 0.217, and 0.170 respectively. This confirms that sequence-based downsampling captures the essential pairwise spatial organization of the protein across scales, with small information losses at the coarsest resolutions.

Per-token vs. per-scale decoder (Section 8.7): An early experiment with a 200M AR transformer and a small MLP diffusion head (20M parameters, as in MAR (Li et al., 2024)) failed to generate reasonable structures, yielding an average sc-RMSD of 16. The paper interprets this as evidence that a per-token decoder (one MLP per residue) cannot capture the global correlations needed to produce a reliable coarse structure at the first scale, which is crucial for subsequent refinement. This motivated the shift to a per-scale transformer-based decoder.

Foldseek cluster diversity (Table 9): The paper explores how the noise scaling parameter γ affects structural diversity, measured by the number of unique Foldseek clusters (TM-score threshold 0.5) among 500 generated structures. Diversity peaks at γ = 0.6 (164 clusters) and remains relatively high across γ = 0.35 to 0.80 (119 to 164 clusters), indicating that the stochasticity parameter provides a tunable tradeoff — lower γ improves designability (as shown in Table 1) at the cost of reduced diversity, while higher γ increases diversity but reduces designability.

Long protein generation (Table 8, Figure 9): After fine-tuning on 26K long protein structures (256–768 residues, curated from PDB with designability filtering), PAR generates proteins at lengths {300, 400, 500, 600, 700}. PAR shows higher designability than Proteina at all lengths: 93% vs. 85% at length 300 (sc-RMSD 1.28 vs. 1.91), 72% vs. 61% at length 400 (sc-RMSD 1.65 vs. 2.70), 52% vs. 49% at length 500 (sc-RMSD 3.19 vs. 4.09), 29% vs. 21% at length 600 (sc-RMSD 6.80 vs. 7.90), and 10% vs. 4% at length 700 (sc-RMSD 11.29 vs. 13.32). Both models degrade at longer lengths, which the authors attribute to the long-tail distribution of the training set (far fewer samples above 500 residues). The paper notes this as a promising direction for future scaling rather than a fundamental limitation of the approach.

Zero-shot motif scaffolding benchmark details (Table 10): The paper reports per-problem success rates for all methods. PAR achieves the highest success rate on 3IXT (6.0% vs. 0.8% for the next-best baseline) and shows competitive performance on 6E6R variants (6%, 3%, 3% vs. Proteina's 5.6%, 41.7%, 71.3% — though Proteina's much higher rates on med/long variants reflect the advantage of task-specific fine-tuning). PAR notably achieves 0% on several problems where baselines achieve non-zero rates (1PRW, 1BCF, 5TPN, 5IUS, 5YUI, 7MRX.60, 4JHW, 4ZYP, 5WN9, 5TRV_short, 5TRV_long), suggesting that certain motif geometries are more challenging for the zero-shot teacher-forcing approach than for fine-tuned diffusion-based inpainting.

Critical Assessment

The experiments collectively support PAR's main contributions, but several important caveats and missing analyses limit the strength of certain claims.

On unconditional generation quality (Claim: PAR captures protein distributions competitively with diffusion models). The evidence in Table 1 provides qualified support. PAR_pdb achieves substantially better FPSD vs. PDB (161.0) than any reported baseline, which is the headline distributional result. However, this comparison has limitations: (a) the PDB-fine-tuned Proteina result is not reported in Table 1, so we cannot determine how much of PAR's FPSD advantage comes from the multi-scale framework vs. simply from PDB fine-tuning (the pre-fine-tune PAR at 400M achieves FPSD 313.9 vs. PDB, which is worse than Proteina's 271.3); (b) the paper reports FPSD vs. both PDB and AFDB, and the relative ordering flips — before fine-tuning, PAR is better on AFDB (296.4 vs. 272.6) but worse on PDB (313.9 vs. 271.3), making it ambiguous which model better captures the "true" protein distribution; (c) the test set for these metrics comprises 5,000 generated structures, which is adequate but not large relative to the diversity of protein fold space. The designability results are clearer: PAR achieves 96.0% pre-fine-tune vs. Proteina's 92.6%, confirming that PAR-generated structures are at least as physically realizable. Overall, the evidence supports that PAR is competitive with state-of-the-art diffusion models, but the claim that it represents a distributional improvement (rather than a different point on the quality-diversity Pareto frontier) depends on which reference distribution is used and whether fine-tuning is applied.

On zero-shot generalization (Claim: PAR supports flexible human-prompted conditional generation and motif scaffolding without fine-tuning). This claim is well-supported for its existence but less so for its practical competitiveness. The prompted generation results (Figure 3, Table 5) convincingly demonstrate that PAR can accept coarse layouts and complete them — this is a genuine capability that single-scale models do not possess without task-specific training. The structural consistency metrics in Table 5 show that the AR transformer actively contributes to this capability (better TM-scores and lower RMSDs than direct input). However, the paper does not benchmark prompted generation against any alternative method (there is no baseline for "generate a protein matching this 16-point layout"), so we cannot assess whether the achieved consistency is good in absolute terms. The TM-scores vs. prompt of 0.60–0.64 (Table 5) indicate moderate but not high structural similarity — by comparison, the "reference" row (downsampling and regenerating the same protein) achieves TM-scores of 0.59–0.61, suggesting that PAR essentially recovers the prompt structure at the level of self-reconstruction fidelity, but this is evaluated only on the test set of 588 structures and may not represent harder generalization cases requiring novel topology completion.

The motif scaffolding results (Table 10) are more mixed. PAR solves 13/24 problems at ≥1% success rate in the zero-shot setting, which is commendable for a method that requires no task-specific training. But the absolute success rates are low — only 4 of 24 problems exceed 5% success rate, and 11 problems achieve 0% — meaning the method is far from practically useful for scaffold design without further development. More critically, the comparison to baselines is confounded by unequal sample budgets (100 for PAR vs. 1000 for baselines) and different evaluation protocols (the paper does not specify whether baseline results used the same success criteria). The paper's framing emphasizes that PAR achieves this zero-shot while baselines require fine-tuning, which is a valid conceptual distinction, but the practical message is that zero-shot PAR is not yet a replacement for fine-tuned motif scaffolding methods on many problems.

On scaling behavior (Claim: PAR exhibits favorable scaling behavior). Figure 5 demonstrates monotonic improvement in FPSD with model size and training steps, which is consistent with the claim. However, the scaling study has notable limitations: (a) only three model sizes are tested, with the largest at 400M parameters — this is in the small-to-medium range by modern standards, and the scaling trends may not extrapolate to larger models; (b) the AR transformer is not scaled (fixed at 60M), so the scaling behavior of the full PAR system (AR transformer + decoder) is not characterized — it is possible that the AR transformer becomes a bottleneck at larger decoder sizes; (c) the training datasets are fixed (0.6M structures, 200K–600K steps), so data scaling is not explored — we cannot distinguish whether the improvements come from model capacity or from the interaction of capacity with the fixed dataset size; (d) the fS(T) metric shows improvement but the paper does not report the C, A, and T components separately at different scales, making it unclear whether the improvement is in quality, diversity, or both. The claim of "favorable scaling behavior" is supported at the scale tested, but the evidence base is thin for a strong scaling claim.

On exposure bias mitigation (Claim: NCL and SS effectively alleviate exposure bias, enabling robust backbone generation). Table 3 provides clear evidence that NCL and SS substantially improve generation quality (sc-RMSD from 2.20 to 1.48). However, the ablation has limitations: (a) it uses a 60M model trained for only 100K steps, which is far from the full training protocol (200K–600K steps on 200M–400M models) — it is unclear whether the same relative improvements would hold at scale; (b) the evaluation uses ODE sampling, which is the most susceptible to exposure bias (SDE sampling provides its own error correction via stochasticity), so the benefit of NCL+SS may be smaller for the SDE-based sampling used in the main results; (c) the paper does not ablate the specific design choices within NCL (how the noise weights w_ncl are sampled, whether per-scale independent sampling matters) or SS (the 0.5 probability, whether teacher-forcing some scales and using predictions for others matters). The techniques are demonstrated to work, but the paper does not characterize how much of the gap between teacher forcing and acceptable generation quality is closed by these methods at the full training scale used for benchmarking.

On sampling efficiency (Claim: PAR achieves a 2.5× sampling speedup compared to single-scale baselines). Table 2 demonstrates this convincingly within the tested configurations. However, the comparison is between PAR with SDE/ODE orchestration and a single-scale model with fixed 400 SDE steps. The paper does not explore whether the single-scale baseline could also be optimized — Proteina might similarly tolerate reduced SDE steps or ODE sampling with appropriate tuning, which would reduce or eliminate PAR's relative speedup. The paper shows in Table 2 that reducing Proteina's SDE steps from 400 to 200 drops designability from 97% to 89%, suggesting limited optimization headroom, but this is only one tested configuration. Furthermore, the speedup numbers (1.96× at length 150, 2.5× at length 200) are specific to the 3-scale PAR with S = {64, 128, 256} — different scale configurations or protein lengths would yield different speedups, and the paper does not characterize the speedup-length relationship beyond these two points.

Missing analyses that would strengthen the paper: (1) A direct comparison of PAR_pdb against PDB-fine-tuned Proteina — this is the most important missing baseline for the unconditional generation claim. (2) Prompted generation benchmarks against inpainting-based methods (e.g., RFDiffusion's motif scaffolding with partial structure constraints), which would contextualize whether PAR's zero-shot prompted generation is competitive with fine-tuned constrained generation. (3) Diversity-quality tradeoff curves (varying γ or sampling steps) for both PAR and Proteina to determine whether PAR's advantages represent a Pareto improvement or a different operating point. (4) Statistical confidence intervals for FPSD and designability, since these metrics are computed over finite sample sets. (5) Generation of structures at lengths not well-represented in the training data to test the generalization limits of the multi-scale framework — the long-protein experiment (Table 8) begins to address this but does not test lengths completely outside the training distribution. (6) An ablation on the number of SDE steps needed at the coarsest scale — the paper notes that 400 steps are needed but does not explore whether this threshold is sharp or gradual, or whether it depends on the coarse scale size (64 in the default configuration).

6. Limitations and Trade-offs

The Current Difficulty Estimation Method Costs More Than the Generation Itself

The assumption or constraint: The compute-optimal allocation policy depends on estimating each prompt's difficulty before choosing a test-time strategy. The paper's method for doing so — generating 2048 samples per question, scoring each with the PRM, and binning based on the average score — is acknowledged in Section 3.2 as a computational burden that is explicitly excluded from the efficiency calculations:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence: In any realistic deployment, the total compute cost is the sum of difficulty estimation plus strategy execution. Since 2048 samples far exceeds the largest test-time budgets studied (256–512 generations), the difficulty estimation step would dominate the total cost, rendering the reported 4× efficiency gains over best-of-N unattainable in practice. A user who naively adopts the compute-optimal framework without addressing this problem would find that the "efficient" allocation actually uses more total compute than simply running best-of-N on every problem. The $4\times$ figure in Figures 4 and 8 should therefore be understood as an upper bound on achievable efficiency, not a realized deployment gain. The gap between this upper bound and practical deployment depends entirely on how cheaply difficulty can be estimated — and the paper provides no evidence that cheap estimation is feasible.

What evidence exists in the paper: The authors flag this explicitly in Section 3.2, noting that the predicted-difficulty approach "still incurs additional computation cost" and that this cost is omitted. The fact that predicted difficulty bins track oracle bins closely (Figures 4 and 8, curves largely overlap) confirms that difficulty can be estimated without ground-truth labels — but this says nothing about whether it can be estimated cheaply. No experiment measures the cost-performance tradeoff of using fewer samples for difficulty estimation (e.g., 16 or 64 rather than 2048), and no lightweight difficulty predictor is trained or evaluated. The paper provides zero evidence about where the difficulty estimation cost vs. accuracy Pareto frontier lies.

Mitigation status: The paper explicitly defers this to future work, suggesting "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). No mitigation is attempted in the current work. The authors frame the current approach as establishing an upper bound, acknowledging that closing the gap between this bound and practical deployment requires solving the cheap-difficulty-estimation problem — a problem they do not solve.

All Results Are Confined to a Single Benchmark on a Single Model Family

The assumption or constraint: Every experiment in the paper — search, revisions, compute-optimal allocation, and the FLOPs-matched comparison — uses the MATH benchmark (500 test questions from Hendrycks et al., 2021) with PaLM 2-S* as the base model. The paper states (Section 4) that the authors "believe this model is representative of the capabilities of many contemporary LLMs," but provides no cross-model or cross-benchmark validation.

The consequence: Multiple aspects of the findings could be model-specific or benchmark-specific in ways that affect the central claims:

The difficulty-dependent behavior may not generalize. The paper's core insight — that beam search hurts easy problems but helps medium ones, that revisions help easy problems but need parallel sampling for hard ones — is characterized relative to PaLM 2-S*'s specific capability profile. A model with different calibration, different error patterns, or a different pass@1 distribution over MATH problems could exhibit qualitatively different difficulty-dependent scaling curves. The five difficulty quintiles are defined by PaLM 2-S*'s pass@1 rates, so a model with higher baseline performance would shift problems into easier bins, potentially changing which strategies are optimal.
MATH is exclusively competition-level symbolic math. The paper's framework — PRM training via Monte Carlo rollouts, revision model training with edit-distance pairing, and the whole concept of difficulty estimation via pass@1 — depends on the existence of verifiable ground-truth answers that can be checked with exact string matching. Many important reasoning tasks (code generation where correctness is functional rather than exact-match, open-ended scientific reasoning, multi-step planning) lack this property. The paper's PRM training pipeline and difficulty estimation method would require fundamentally different approaches in those domains.
The revision model's behavior may depend on PaLM 2-S's specific in-context learning capabilities.* The revision training data construction — pairing incorrect answers with correct ones and training the model to revise — relies on the base model being able to condition on its own outputs. Different model families show substantially different sensitivity to in-context information.

What evidence exists in the paper: No cross-model experiments exist. No results on any benchmark other than MATH are reported. The authors acknowledge this implicitly by framing their model choice as "representative" (Section 4), but this is an assertion rather than an empirical finding. The paper also notes in Section 8 that extending to other domains is future work.

Mitigation status: None. This is a deliberate scoping decision — the paper is a first systematic study of compute-optimal test-time scaling and chooses depth (thorough analysis on one model-benchmark pair) over breadth. However, this means the main findings should be treated as contingent on both the model and the benchmark, not as universal properties of test-time compute scaling.

The $14\times$ Larger Model Baseline Is Weakened by Non-Optimal Pretraining and the Absence of Test-Time Compute

The assumption or constraint: The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately $14\times$ more parameters, trained with fixed data (following the LLaMA paradigm of scaling parameters only) rather than compute-optimal pretraining where both parameters and data are scaled (following the Chinchilla paradigm). Moreover, the larger model uses only greedy decoding with no test-time compute augmentation of its own.

The authors explicitly acknowledge this choice (Section 7):

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence: Both weaknesses bias the comparison in favor of test-time compute. A Chinchilla-optimal $14\times$ larger model (with both parameters and training tokens scaled appropriately) would likely outperform a parameter-only-scaled model, making the pretraining baseline stronger. And giving that larger model even a modest test-time compute budget — say, majority voting with 8 or 16 samples, which is standard practice in many deployments — would create a significantly more demanding comparison. The current baseline effectively assumes that the larger model's additional pretraining compute cannot be complemented by any inference-time strategy, which is unrealistic for most production systems where some form of sampling or verification is applied regardless of model size.

The quantitative consequence is that the reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy-medium questions at $R \ll 1$ (Figure 1, top-right bar chart) — are upper bounds that would likely shrink against a properly optimized larger model. We do not know by how much, because the paper provides no ablations varying the larger model's test-time compute budget or its pretraining data scaling.

What evidence exists in the paper: The paper is transparent about this limitation, stating it explicitly in Section 7. The three values of $R$ (0.16, 0.79, 22) capture different inference-to-pretraining ratios, which is useful for understanding when the test-time-vs-pretraining tradeoff matters. But within each $R$ regime, the larger model baseline is fixed at greedy decoding with parameter-only scaling. No ablation tests how the comparison changes if the larger model is given best-of-N, majority voting, or any other test-time strategy.

Mitigation status: The paper acknowledges this as future work (Section 7) but does not address it in the current experiments. Readers should interpret the FLOPs-matched results as a comparison between test-time compute with a small model and pure pretraining with a larger model without any test-time augmentation — a comparison that isolates the pretraining-vs-inference tradeoff but does not represent how the two approaches would compete in a setting where both are optimized.

Test-Time Compute Cannot Help on Problems Where the Base Model Has Near-Zero Pass@1

The assumption or constraint: The entire compute-optimal framework — both PRM search and iterative revisions — operates by finding, refining, or selecting among candidate solutions generated by the base model. If the base model produces correct solutions at a negligible rate, no test-time strategy can help, because there are no correct candidates to find or refine. The paper acknowledges this explicitly (Section 7 takeaway box):

"Test-time compute can amplify existing capability but does not create it from nothing."

The consequence: For the hardest problems (difficulty bin 5 in the paper's taxonomy), test-time compute provides essentially zero benefit regardless of budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets up to 256 generations. In Figure 7 (right), bin 5 revision accuracy is roughly 2–3% irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is flat near 0–5% for all $R$ values, and is consistently below the larger model's performance. The paper's approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution.

This limitation is not just an empirical observation — it is a structural constraint of the framework. Because the proposal distribution (base model) is the sole source of candidate solutions, and test-time compute only re-ranks or refines those candidates, the maximum achievable accuracy is bounded by the base model's pass@1 rate times the effectiveness of the verifier at identifying correct solutions. When pass@1 is effectively zero, this product is zero regardless of compute allocation. This contrasts with pretraining, which can qualitatively expand the set of problems the model can solve by learning new capabilities.

What evidence exists in the paper: The difficulty-bin analyses throughout the paper consistently show bin 5 as flat and near-zero. Figure 3 (right) is the clearest visualization: all methods — best-of-N, beam search, majority voting — are indistinguishable on bin 5 across all budgets. Figure 9 shows that even in the most favorable regime ( $R \ll 1$ , where test-time compute gets a large budget), bin 5 performance under compute-optimal scaling remains far below the larger model's greedy performance. The paper is notably transparent about this limitation; it is not hidden or downplayed.

Mitigation status: The paper acknowledges this boundary explicitly (Section 7) and does not attempt to overcome it. The boundary serves as a useful characterization of where test-time compute is applicable — problems within the base model's rough capability range — but it also means the method is fundamentally complementary to pretraining, not a replacement for it. For any problem distribution with a substantial fraction of genuinely novel or hard problems, pretraining remains necessary. The paper's framing suggests this is a feature (the approach tells you when to invest in pretraining versus test-time compute), but it is a limitation for anyone hoping test-time compute alone would suffice.

No Error Bars, Confidence Intervals, or Statistical Tests Are Reported

The assumption or constraint: All quantitative results in the paper — accuracy percentages, FPSD scores, sc-RMSD values, and the comparisons between strategies — are reported as point estimates from a single evaluation run on a fixed test set (500 MATH questions or 500–5000 generated protein structures). The paper does not report confidence intervals, standard errors, or statistical significance tests for any comparison.

The consequence: The paper makes precise comparative claims — e.g., "compute-optimal scaling nearly outperforms best-of-N using up to $4\times$ less test-time compute" (Section 5.3), "beam search significantly outperforms best-of-N at low generation budgets" (Section 5.3) — without quantifying the uncertainty around those claims. This matters because:

The test set is small relative to the complexity of the strategy space. The 500-question MATH test set is split into five difficulty quintiles of ~100 questions each. With two-fold cross-validation, strategy selection is based on ~50 questions per fold per bin. The optimal strategy for a given bin-budget pair is a discrete choice among several candidate strategies (different search algorithms, beam widths, sequential-to-parallel ratios). With ~50 samples, the variance in estimated accuracy for each strategy could be substantial, and the "optimal" strategy might not be reliably distinguishable from the second-best. The paper provides no way to assess whether the reported gains are statistically reliable at this sample size.
The FLOPs-matched comparison involves ratios of relatively small numbers. For the hardest difficulty bin (bin 5), accuracy rates are in the 1–5% range. A difference of a few percentage points — which could determine whether test-time compute "outperforms" pretraining — might be within sampling noise when evaluated on ~100 questions. Without error bars, there is no way to know.
The revision model's sequential-to-parallel ratio sweep (Figure 7) involves comparing many configurations with overlapping performance. Are the apparently optimal ratios at intermediate values meaningfully different from the extremes, or are the curves flat within noise? The figure provides no way to tell.

What evidence exists in the paper: None. This is an absence, not a finding. The paper's evaluation protocol is standard for the protein structure generation literature (Geffner et al., 2025 similarly reports point estimates), but the LLM test-time compute community generally expects confidence intervals, particularly when making comparative claims about efficiency improvements. The paper does not discuss this limitation.

Mitigation status: Not addressed. The cross-validation protocol for strategy selection (Section 3.2) mitigates overfitting the policy to the test set, but it does not provide uncertainty quantification around the reported metrics. Reporting confidence intervals would require either multiple training runs (to capture model training variance) or bootstrap resampling of the test set (to capture evaluation variance), neither of which is performed.

Sequential Revisions and PRM Search Are Studied Independently, Not Combined

The assumption or constraint: The paper studies two complementary axes of test-time compute — PRM-guided search (Section 5) and iterative revisions (Section 6) — as separate, parallel pipelines. The compute-optimal allocation selects between these approaches per difficulty bin, but never combines them: the revision model's outputs are never used as candidates for beam search, and the PRM is never used to guide which revisions to pursue or when to stop revising.

The authors explicitly acknowledge this gap (Section 8):

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence: The reported results represent a lower bound on what a fully integrated system could achieve, because the two mechanisms have complementary strengths. Revisions improve the proposal distribution — the revision model generates higher-quality candidates in the first place, especially on easy problems where local refinement helps. PRM search improves candidate selection — beam search explores the solution space more efficiently than random sampling, especially on medium problems where the base model has partial capability but needs guidance to find correct solutions.

A combined system might, for instance, use the revision model as the proposal distribution within beam search (each beam step conditions on previous revisions), or use the PRM to decide mid-chain whether to continue revising or restart with a different approach. The paper's finding that the optimal strategy varies by difficulty (beam search on medium problems, revisions on easy problems) suggests that a combined approach might achieve gains on problems where neither pure search nor pure revisions are optimal — particularly on the medium-hard boundary where both exploration and refinement are needed.

The practical consequence is that the paper's headline $4\times$ efficiency gain (Section 1) is achieved by choosing between search and revisions, not by combining them. A system that combines both could potentially achieve larger gains or extend the range of problems where test-time compute helps (pushing into harder difficulty bins where neither approach alone is effective).

What evidence exists in the paper: The difficulty-dependent analysis of both search (Figure 3, right) and revisions (Figure 7, right) shows that the two mechanisms have different optimal operating regimes — search excels on medium problems, revisions excel on easy problems. This is indirect evidence that combining them could be beneficial, but no experiment tests this directly. Section 8 explicitly flags this as future work.

Mitigation status: Not addressed in the current work. The paper frames this as a natural next step (Section 8) but provides no experimental evidence about how the combination would perform, what architecture would be needed, or whether the exposure bias challenges that affect the revision model would interact with the verifier over-optimization challenges that affect beam search in unexpected ways. This is a significant gap for practitioners who might want to build a maximally performant system — the paper tells them which strategies work in which regimes, but not how to integrate them.

7. Implications and Future Directions

How This Work Changes the Landscape

PAR's primary impact on the protein design field is conceptual reframing rather than a paradigm overthrow: it demonstrates that autoregressive modeling — the dominant paradigm in language and increasingly in vision — can be productively applied to protein backbone generation, but only if you change what "autoregressive" means. The specific reframing is from token-level to scale-level autoregression: instead of generating one residue at a time (which conflicts with bidirectional protein biochemistry) or one discretized token at a time (which loses structural fidelity), PAR generates entire structural representations at progressively finer resolutions, each conditioned on all coarser representations. This dissolves the two objections that had kept AR models out of protein backbone generation — unidirectional dependency and discretization loss — without sacrificing the AR properties (scalability, zero-shot generalization) that make AR models attractive in other domains.

The paper's most lasting contribution may be the generality of this reframing. Section 5 states that PAR "provides a more general framework, incorporating flow-based models as a special case when restricted to a single scale." This is not marketing — it is a structural truth. If you configure PAR with n = 1 (only the full-resolution scale), the AR transformer produces a single conditioning vector, the flow decoder denoises it with flow matching, and the system is architecturally identical to Proteina (the single-scale flow model that serves as PAR's primary baseline). The multi-scale extension adds autoregressive structure without modifying the underlying generative machinery — the flow decoder architecture, the flow matching loss, the SDE/ODE sampling, and self-conditioning all remain unchanged. This means that any improvement to single-scale flow models (better architectures, better sampling, better loss functions) immediately benefits PAR, and vice versa — insights from PAR's multi-scale training (exposure bias mitigation, scale-dependent compute allocation) could be back-ported to single-scale models.

This collapses the false dichotomy between "diffusion/flow models" and "autoregressive models" for protein design. PAR demonstrates that you can have both: the distributional fidelity and physical realizability of flow matching (Table 1: 96.6% designability, FPSD 161.0 vs. PDB after fine-tuning), and the zero-shot task flexibility and interpretable generation process of autoregressive models (Figures 3 and 4: prompted generation with 16 points, motif scaffolding without fine-tuning). Researchers no longer need to choose between these property sets — the multi-scale AR framework provides a path to both simultaneously.

The paper also shifts the exposure bias conversation for structured generation. In language and image AR modeling, exposure bias is often treated as a manageable annoyance — techniques like scheduled sampling provide modest improvements, but the problem is rarely a showstopper. PAR's experience (Table 3) reveals that for structured, continuous domains where errors compound geometrically across scales, exposure bias is a first-order barrier: naive teacher forcing produces sc-RMSD of 2.20 Å vs. 1.48 Å with NCL+SS — a ~50% degradation that would render the method non-competitive. This finding should recalibrate expectations for anyone attempting multi-scale AR in other structured domains (3D molecular generation, mechanical CAD, robotic trajectory planning). The paper's specific mitigation (noisy context learning + scheduled sampling) is not novel in its components, but the diagnosis that the severity of exposure bias scales with structural precision requirements and data limitations is a conceptual contribution that will influence how future work approaches this problem.

Finally, the paper provides a unifying explanation for conflicting results in the nascent protein AR literature. Gaujac et al. (2024) showed that token-based AR protein generation is possible but limited. The field might have concluded that AR modeling is fundamentally ill-suited to proteins. PAR shows that the limitation was not AR modeling per se, but token-level AR modeling with discretization and unidirectional residue ordering. By changing the unit of autoregression, the limitations vanish. This is analogous to how VAR (Tian et al., 2024) resolved the tension between AR image generation and 2D spatial structure by introducing next-scale prediction — PAR extends this resolution to 3D protein structures, confirming that the scale-level reframing is a broadly applicable pattern for AR modeling of structured data.

Follow-Up Research This Work Enables

Scaling PAR to billion-parameter decoders and larger datasets. The paper's scaling study (Figure 5) tests decoder sizes up to 400M parameters on 0.6M training structures for up to 600K steps, showing monotonic improvement in FPSD and sc-RMSD but without evidence of saturation. The natural next step is to scale to 1B+ parameter decoders on multi-million-structure datasets (e.g., the full AFDB with tens of millions of predicted structures, or the ESM3 training corpus). The specific question is whether the favorable scaling trends in Figure 5 continue, or whether bottlenecks emerge — particularly whether the 60M AR transformer (which the paper shows does not benefit from scaling due to exposure bias overfitting; Table 12) becomes the limiting factor at very large decoder sizes. A strong follow-up would measure FPSD, designability, and sc-RMSD at 1B, 3B, and 7B decoder parameters, and test whether scaling the AR transformer jointly becomes necessary beyond some decoder size threshold. The paper's finding that exposure bias intensity scales with AR model size (Table 12: 400M AR + 60M decoder underperforms 60M AR + 60M decoder) suggests that larger AR transformers would need improved mitigation strategies — perhaps adversarial training of the AR module, or training on on-policy rollouts from the decoder rather than ground-truth context.

Combining PAR with residue-level autoregressive sequence-structure co-generation. PAR generates backbone structures conditioned on scales, but does not generate amino acid sequences. A natural extension is to interleave structure and sequence autoregression: at each scale, first generate the structural representation xⁱ (using the flow decoder conditioned on zⁱ from the AR transformer), then generate a sequence representation sⁱ at that scale conditioned on xⁱ and prior scales, and feed sⁱ back as additional context for the next scale's conditioning. This would produce a unified sequence-structure generative model that captures the co-evolutionary relationship between sequence and structure across scales. The paper's framework makes this tractable because the AR transformer already accepts multi-modal input (3D coordinates at multiple scales); extending to include sequence embeddings would only require modifying the input concatenation and training data. A concrete experiment: pre-train PAR on AFDB structures with their native sequences, then measure whether the co-generation model produces higher-designability structures (because the sequence context constrains the structure generation) and whether the generated sequences fold to the generated structures with higher recovery rates than sequences designed post-hoc by ProteinMPNN. The zero-shot generalization properties should also extend: a co-generation model could accept partial sequence constraints (e.g., a fixed binding motif sequence) and complete both the sequence and structure autoregressively.

Training a lightweight difficulty estimator for adaptive scale allocation. The paper demonstrates (Table 2, Figure 7) that PAR can allocate different sampling strategies (SDE vs. ODE, different numbers of steps) at different scales, achieving a 2.5× speedup. But the allocation is fixed — 400 SDE steps at scale 1, 2 ODE steps at scales 2–3, regardless of the protein. A learnable controller could decide adaptively: after generating the coarse structure x¹, assess its quality (e.g., via the flow model's own likelihood estimate or a learned quality predictor) and decide whether to invest more SDE steps at scale 2 or proceed with ODE. The specific experiment: train a small MLP that takes the AR transformer's conditioning embedding z² and the coarse structure x¹ as input and predicts whether additional SDE sampling at scale 2 would improve the final designability. Train this predictor on rollouts from PAR with varying sampling configurations, then evaluate whether the adaptive policy achieves better quality-compute tradeoffs than the fixed S/O/O policy in Table 2. The paper's evidence that the first-scale SDE step is critical for topology formation (reducing SDE steps at scale 1 degrades designability; Figure 7) and that later-scale ODE suffices once topology is established suggests a natural adaptive strategy: invest heavily in coarse-scale SDE until a quality criterion is met, then switch to efficient ODE refinement.

Extending zero-shot motif scaffolding with iterative refinement at the motif boundaries. Table 10 shows that PAR achieves zero-shot motif scaffolding success on 13/24 benchmark problems, but with low absolute success rates (only 4 problems above 5%) and 11 problems at 0%. The teacher-forcing approach — superimposing the ground-truth motif coordinates at each scale before generating the next — can create discontinuities at motif-scaffold boundaries because the scaffold is generated without awareness of the motif's exact coordinates until post-hoc alignment. A more sophisticated zero-shot approach would use the flow decoder's ability to perform conditional generation: at each scale, mask the motif residues and use the flow decoder to infill the scaffold residues conditioned on the motif as fixed context, analogous to image inpainting with diffusion models. This would require no additional training — the flow decoder architecture already supports per-residue conditioning — and would produce smoother motif-scaffold junctions. The experiment: re-run the 24-problem benchmark with masked conditioning instead of post-hoc superimposition, measure success rates (scRMSD ≤ 2Å, motifRMSD ≤ 1Å, pLDDT ≥ 70, pAE ≤ 5) with 100 and 1000 samples, and compare to the teacher-forcing approach in Table 10. This would determine whether the zero-shot performance gap relative to fine-tuned methods (particularly on problems like 6E6R_med where Proteina achieves 41.7% vs. PAR's 3%) is due to the scaffolding mechanism or fundamental limitations of zero-shot generalization.

Stress-testing the multi-scale framework with out-of-distribution protein topologies. The paper's training data (AFDB representative set, 32–256 residues, pLDDT > 80) covers a specific distribution of protein folds, lengths, and topologies. The multi-scale autoregressive framework makes a specific inductive bias claim: that generating coarse topology first and refining it produces coherent structures regardless of whether the topology is in-distribution. A stress test would evaluate PAR on fold classes, symmetries, and topologies that are absent from or underrepresented in AFDB. For example: generate circularly permuted variants of known folds (where the N-to-C sequence order is rearranged but the 3D structure is similar), generate symmetric oligomers (dimers, trimers, cyclic symmetries) by providing symmetric coarse prompts, or attempt to generate completely novel folds by specifying coarse layouts that correspond to no known protein topology. The key measurements: (1) Does PAR produce designable structures for out-of-distribution topologies, or does it "snap back" to in-distribution folds during fine-scale refinement? (2) Does the AR transformer's scale-attention pattern (Figure 6) change for out-of-distribution topologies, indicating that it relies on learned priors rather than general structural reasoning? (3) Can increasing the SDE stochasticity at coarse scales (higher γ) drive exploration of novel topologies, and does this come at the expected cost of reduced designability (per the γ-designability tradeoff in Table 1)? This stress test would clarify whether PAR's generation is fundamentally interpolative (sampling from the learned distribution) or can extrapolate to novel structural regimes — a critical distinction for de novo protein design applications.

Developing robust exposure bias mitigation for larger AR transformers. The paper's finding that scaling the AR transformer from 60M to 400M degrades performance (Table 12: designability drops from 92.6% to 87.8%) is attributed to exposure bias — larger models overfit ground-truth context more aggressively. The current mitigation (NCL + SS) was sufficient for a 60M AR transformer but apparently insufficient at 400M. A targeted follow-up would explore more aggressive mitigation strategies specifically designed for large AR modules in structured domains. Candidates include: (1) Adversarial context training: train the AR transformer to produce conditioning zⁱ that is robust to worst-case perturbations of the input context, rather than just random Gaussian noise (NCL). This could be implemented by adding a small adversarial perturbation to the context that maximizes the flow matching loss, then training the AR transformer to minimize loss under this perturbation. (2) On-policy context generation: during training, replace ground-truth prior scales with rollouts from the model itself (not just one-step predictions as in scheduled sampling, but full multi-step generations with the current model parameters), creating a training distribution that exactly matches inference. This is computationally expensive but directly addresses the root cause. (3) Two-stage training: pre-train the AR transformer with teacher forcing for fast convergence, then fine-tune with progressively more aggressive context corruption (increasing NCL noise, increasing SS probability) to gradually close the train-test gap. The experiment would measure whether any of these strategies enables a 400M AR transformer to outperform the 60M AR transformer (reversing the trend in Table 12), and at what computational cost.

Practical Applications and Downstream Use Cases

Rapid prototyping of protein backbones with user-specified coarse constraints. PAR's zero-shot prompted generation (Figure 3) enables a workflow where a structural biologist specifies a rough 3D layout (as few as 16 points in 3D space, which could be sketched in a molecular visualization tool or derived from a low-resolution cryo-EM map) and PAR generates complete, designable backbones that respect the specified global architecture. The TM-score vs. prompt of 0.60–0.64 (Table 5) indicates moderate-to-strong structural consistency with the 16-point prompt, and the generated structures are highly designable (96.0% at 400M, Table 1). This is immediately useful for scaffold design where the overall fold is constrained but local details are flexible — for example, designing a new enzyme with a specified active site geometry but a novel scaffold architecture. The zero-shot nature means no retraining is needed for each new design target, making rapid iteration feasible. The ability to increase prompt resolution (e.g., 32 points for finer control) provides a tunable knob for constraint strength.

Scaffold diversification for lead optimization in protein therapeutics. The zero-shot motif scaffolding capability (Figure 4, Table 10) enables generating diverse protein scaffolds around a fixed functional motif without training a task-specific model. In a drug discovery setting, a lead candidate protein with a known binding interface (the "motif") needs to be diversified — same binding function, different scaffold for improved stability, expressibility, or reduced immunogenicity. PAR can take the binding interface coordinates as motif input and generate hundreds of scaffold variants, each preserving the binding residues while introducing novel structural elements elsewhere. The examples in Figure 4 show that PAR introduces genuine structural novelty (new secondary structure elements, different loop geometries) rather than minor variations on the original scaffold. The 100-sample evaluation in Table 10 is small relative to industrial needs but demonstrates feasibility — scaling to thousands of samples per motif would be straightforward and would increase the probability of finding scaffolds with desired properties. The fold diversity data (Table 9: 119–164 clusters depending on γ) provides a tunable diversity-quality tradeoff.

Computationally efficient generation for high-throughput virtual screening. The SDE/ODE orchestration (Table 2) provides a 2.5× sampling speedup over single-scale models at equivalent designability, and this advantage grows with protein length (because the coarse-scale cost is fixed at 64 residues while the fine-scale savings scale quadratically). For large-scale virtual screening pipelines that need to generate and evaluate millions of protein backbones — such as designing binders against a panel of targets, or exploring sequence space for a given fold — this speedup translates directly to reduced GPU-hours and faster iteration cycles. At length 200, PAR generates 100 designable structures in 68 seconds vs. 170 seconds for Proteina (Table 2), meaning a pipeline that needs 100,000 structures would require approximately 19 GPU-hours with PAR vs. 47 GPU-hours with the single-scale baseline. The speedup is likely larger for longer proteins (the scaling advantage grows with length), though this is not directly measured beyond length 200. The γ parameter provides a further efficiency knob: lower γ improves designability at the cost of diversity (Table 1, Table 9), so pipelines that need highly designable structures can trade diversity for quality without retraining.

Long-protein design with improved success rates. The long-protein experiment (Table 8) shows that PAR maintains higher designability than Proteina at extended lengths: 93% vs. 85% at length 300 (sc-RMSD 1.28 vs. 1.91), 72% vs. 61% at length 400, and a 2.5× advantage at length 700 (10% vs. 4%). While the absolute designability drops at extreme lengths (a limitation of the training data distribution; Figure 9 shows sparse samples above 500 residues), the relative advantage suggests that PAR's multi-scale approach is particularly beneficial for long proteins where establishing a reliable global topology is critical. For practitioners designing large multi-domain proteins or extended repeat proteins, PAR offers a higher probability of generating designable backbones on the first attempt, reducing the number of samples that must be generated and experimentally validated. The advantage is likely to increase with further scaling of the long-protein training data, as the authors note this as a promising direction.

When to Prefer This Method

The paper positions PAR explicitly against single-scale diffusion and flow models for protein backbone generation, and the tradeoffs are clear enough to warrant decision guidance:

Prefer PAR when you need zero-shot conditional generation without task-specific training. The defining capability of PAR relative to all baselines in Table 1 is that prompted generation (Figure 3) and motif scaffolding (Figure 4, Table 10) work out of the box with a pretrained model. If your protein design workflow involves frequently changing constraints — different motifs, different coarse layouts, different symmetry requirements — PAR's zero-shot flexibility eliminates the need to retrain or fine-tune for each new conditioning type. The baselines (Proteina, Genie2, RFDiffusion, FrameFlow) require task-specific training for motif scaffolding; PAR achieves competitive success rates (13/24 benchmark problems at ≥1%) without any. The tradeoff is that zero-shot performance lags behind fine-tuned methods on some problems (Table 10), so for a fixed, high-stakes motif scaffolding task where maximum success rate matters, fine-tuning a single-scale model may still be preferable.
Prefer PAR when sampling speed matters and you generate long proteins. The SDE/ODE orchestration (Table 2) provides a 2.5× speedup at length 200 that increases with protein length. If your pipeline generates many structures (thousands to millions) or operates under latency constraints (e.g., interactive design tools), PAR's ability to spend computation only where it matters — stochastic SDE at the coarse scale, efficient ODE at fine scales — directly reduces cost and wall-clock time. Single-scale models apply uniform computation across all residues and all denoising stages, which is simpler but less efficient.
Prefer PAR when interpretability of the generation process is valuable. The coarse-to-fine sculpting process (Figure 2) and the attention map analysis (Figure 6) provide visibility into how structures are constructed — coarse topology first, local detail later. This is useful for debugging generation failures (if the coarse scale is wrong, the fine scales will be wrong regardless of refinement quality), for interactive design (a user can inspect and modify the coarse layout before committing to expensive fine-scale refinement), and for scientific understanding of what the model learns about protein folding hierarchies. Single-scale diffusion models provide black-box generation with no analogous interpretability.
Prefer single-scale flow/diffusion models when unconditional distributional fidelity is the sole objective. Before PDB fine-tuning, PAR (400M) and Proteina (400M) are comparable on designability (96.0% vs. 92.6%) but diverge on FPSD — PAR is better against AFDB (296.4 vs. 272.6) while Proteina is better against PDB (271.3 vs. 313.9). This ambiguity means neither model unambiguously dominates for distributional matching. The additional complexity of PAR (AR transformer, exposure bias mitigation, scale-dependent sampling configuration) may not be justified if you only need to sample from the protein structure distribution and don't need zero-shot conditioning, interpretability, or sampling efficiency. For a simple unconditional generation pipeline, a trained Proteina model with 400 SDE steps is a well-understood, reliable baseline that requires fewer design decisions.