Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers

ArXiv: 2311.10642

🎯 Pitch

Shallow feed-forward networks can successfully replace self-attention in Transformers (within 1–2 BLEU points), but fail catastrophically when replacing cross-attention—collapse occurs because cross-attention’s asymmetric query-key alignment is fundamentally harder to distill into a fixed-weight function. This reveals a sharp functional boundary between self- and cross-attention, challenging the assumption that attention is a monolithic mechanism.

---

1. Executive Summary

This work studies whether shallow feed-forward networks can replace attention mechanisms in the original Transformer model, using the IWSLT2017 translation benchmark with a reduced-embedding (128-dim) Transformer as the teacher. The authors propose four replacement approaches operating at different levels of abstraction — ALR (replacing the multi-head attention block while keeping residual connections), ALRR (replacing attention plus the residual connection), ASLR (replacing each attention head with a separate feed-forward network), and ELR (replacing the entire encoder layer) — all trained via knowledge distillation from the original Transformer's intermediate activations. The ALR approach at its largest size achieves BLEU scores within ~1–2 points of the baseline across all four translation pairs (e.g., 0.252 vs. 0.257 baseline on English-German), demonstrating that feed-forward networks can successfully emulate self-attention behavior, establishing however that this substitution is viable only for self-attention — cross-attention replacement collapses performance to near-random levels (BLEU ~0.12 at best vs. ~0.29 baseline), revealing a sharp boundary in what simple feed-forward networks can capture about attention's functionality.

2. Context and Motivation

The Core Problem: Can Attention Be Replaced by Something Simpler?

This paper tackles a question that is both conceptually fundamental and practically consequential: does the Transformer architecture actually need attention, or can much simpler components achieve the same functionality? The question arises from a tension in how we understand the Transformer's success. Since Vaswani et al. (2017), attention mechanisms have been treated as the indispensable innovation that enables Transformers to model long-range dependencies in sequential data — allowing each element to attend to every other element in a sequence, something prior architectures like RNNs and LSTMs struggled to do without severe computational bottlenecks or vanishing gradients. But is attention truly necessary as a computational primitive, or is it simply a convenient architectural choice that happened to work well? This paper tests the extreme version of that question: what if we strip attention out entirely and replace it with a plain shallow feed-forward network?

This is not a question about inventing a new architecture with competitive advantages. The authors are explicit about this in Section 1:

"While it does not introduce a competitive advantage over established methods, it offers a conceptual analysis of existing techniques and potential alternatives."

Rather, this is an analysis paper that probes the boundaries of what attention contributes to the Transformer. By showing what a simple replacement can and cannot do, the work illuminates which aspects of attention are genuinely essential and which are incidental.

Why This Question Matters: Theoretical and Practical Stakes

The motivation operates on two levels:

Theoretical significance — understanding what attention does. If a shallow feed-forward network can learn to replicate the input-output mapping of a multi-head attention block through knowledge distillation, it suggests that attention's contribution — at least for the specific tasks and scales tested — can be captured by a fixed, non-dynamic computation. This is surprising because attention is fundamentally content-dependent: the weights assigned to different input positions depend on the input itself (via query-key dot products), making it a dynamic, data-dependent routing mechanism. A feed-forward network, by contrast, applies the same learned weights to every input. If the latter can approximate the former, then for this particular dataset and model scale, the dynamic routing aspect of attention may not be contributing much beyond what a sufficiently expressive static mapping can provide.

This has implications for how we think about neural architecture design. The success of feed-forward replacements suggests that much of what attention accomplishes in practice may be a form of learned positional pattern matching — recognizing and transforming specific patterns in the input sequence that recur across examples — rather than genuine dynamic reasoning about relationships between tokens. If true, this would partially demystify attention's effectiveness and point toward simpler architectural alternatives.

Practical significance — the cost of attention. While the paper's replacements are actually larger in parameter count than the original attention blocks (see Table 1: 320K–41M parameters for a single replacement FF network vs. 60K parameters for the original attention layer), the conceptual demonstration that attention can be replaced has practical implications. Attention's quadratic $O(n^2)$ complexity in sequence length remains a fundamental bottleneck for processing long sequences. If future work can compress these feed-forward replacements — through pruning, quantization, or improved training — or find architectures that match their performance at lower cost, the result could be sequence models that are more efficient without sacrificing quality. The paper is a proof of concept that the attention operation itself is not strictly necessary, opening the door for alternative designs.

Additionally, the work speaks to the optimization landscape of neural networks. A key finding (Section Discussion) is that:

"These conclusions also point out the deficiencies of the current optimization methods, which are not able to train these 'attentionless Transformers' from scratch but need more advanced techniques, such as knowledge distillation to converge into desired parameter configurations."

In other words, the replacement networks can represent the target function (since they learn it through distillation) but cannot discover it through standard end-to-end training. This is a subtle but important point: the failure is not one of representational capacity but of optimization. The attention mechanism may serve as an inductive bias that guides optimization toward good solutions, rather than as the only architecture capable of representing the necessary computations. This reframes attention from a representational necessity to an optimization aid — a perspective with implications for how we design future architectures and training procedures.

Prior Work: The Knowledge Distillation Lineage

This paper does not emerge from a vacuum. It explicitly positions itself within a research lineage that tests whether complex neural architectures can be emulated by simpler ones through knowledge distillation:

Ba and Caruana (2014) asked "Do Deep Nets Really Need to be Deep?" and showed that shallow feed-forward networks could match the performance of deep convolutional networks on CIFAR-10 when trained to mimic the deep model's outputs. The key insight — which this paper directly inherits — is that distillation compresses the function learned by a complex model into a simpler architecture, and if the simpler architecture can achieve comparable accuracy, then the complexity of the original model was not strictly necessary for that task.

Urban et al. (2017) extended this logic with "Do Deep Convolutional Nets Really Need to be Deep and Convolutional?" They demonstrated that even the convolutional inductive bias could be discarded: shallow fully-connected networks, trained via distillation from deep CNNs, could achieve competitive results on image classification. This is the most direct precursor to the current work, establishing the template of using distillation to test whether a specialized architectural component (convolutions, and now attention) is genuinely required.

Hinton, Vinyals, and Dean (2015) provided the distillation framework itself: training a "student" model to match the softened output distribution of a "teacher" model, transferring not just the hard labels but the richer information contained in the teacher's confidence across classes.

Where this paper differs from its predecessors is in the target architecture and the specificity of the investigation. Rather than compressing an entire model, the authors perform a surgical replacement of specific components — self-attention in the encoder, self-attention in the decoder, and cross-attention — allowing an ablation-level analysis of which attention components are necessary and which are fungible. This component-by-component approach is more granular than prior work, which typically replaced entire models wholesale.

Where Prior Approaches Fall Short — and How This Paper Fills the Gap

No prior work tested attention replacement specifically. The Ba & Caruana and Urban et al. papers targeted convolutional architectures for vision tasks. The Transformer's attention mechanism is fundamentally different from convolution: it is content-dependent rather than spatially fixed, and it operates on sets rather than grids. Whether the distillation-to-simpler-architecture strategy transfers to attention-based sequence models was an open empirical question. This paper provides the first systematic test.

Prior distillation work focused on model compression, not architectural necessity. The typical goal in the distillation literature is to produce a smaller, faster model for deployment. The goal here is different: the replacement networks are often larger than the attention blocks they replace (Table 1), so compression is not the point. The point is to test whether attention is architecturally necessary — a different and more fundamental question.

No existing work distinguished between different attention types in the Transformer. The Transformer contains three distinct attention mechanisms: encoder self-attention (bidirectional context), decoder self-attention (causal/masked context), and decoder cross-attention (attending from decoder to encoder outputs). These have different computational structures and different functional roles. By testing replacements for each type separately, the paper reveals that they are not equally replaceable — self-attention is learnable by feed-forward networks, while cross-attention largely resists the substitution. This distinction would be invisible in a whole-model replacement study and represents a genuinely new empirical finding about the nature of attention.

How the Paper Positions Itself

The paper positions itself as an architectural analysis rather than a proposal for a new model or method. The framing in Section 1 is explicit:

"We aim to assess the extent to which standard shallow feed-forward networks can model attention mechanisms by substituting key attention components with feed-forward networks trained to replicate their behavior."

The verb "assess" signals the paper's analytical stance. The contribution is not "here is a better Transformer" but rather "here is what happens when you try to remove attention from the Transformer, and what that tells us about attention itself."

The paper also positions itself as contributing to a broader discourse about architectural specialization vs. generalization. If shallow feed-forward networks — the most generic neural building block — can replicate the behavior of attention, then attention's value may lie less in its computational uniqueness and more in its role as an optimization-friendly inductive bias. This framing connects to ongoing debates in deep learning about whether specialized architectures are genuinely necessary or whether generic architectures with sufficient scale and better optimization could eventually subsume them.

The authors are careful not to overclaim. They acknowledge that the replacements come at a parameter cost (Table 1), that cross-attention replacement fails (Table 3: decoder cross-attention XS through L sizes achieve BLEU scores of 0.035–0.130 across datasets, far below the 0.257–0.324 baseline), and that the fixed-size input requirement eliminates the Transformer's native ability to handle variable-length sequences without padding. These limitations are presented not as failures of the approach but as boundary conditions that sharpen the analysis: self-attention is replaceable because its functionality can be captured by a fixed mapping (at least for this dataset), while cross-attention may involve genuinely dynamic, content-dependent computations that resist static approximation.

3. Technical Approach

This is primarily an empirical analysis paper whose core idea is that the multi-head attention mechanism in Transformers can be functionally replaced by simple shallow feed-forward networks trained via knowledge distillation, at least for self-attention tasks, suggesting that attention's role may be more about providing beneficial optimization landscapes during end-to-end training than about being the only architecture capable of representing the necessary sequence transformations.

3.1 Reader Orientation

What is the system? The system is a modified Transformer model for sequence-to-sequence language translation where specific attention components—the multi-head self-attention and cross-attention blocks—are surgically removed and replaced with shallow (one-hidden-layer) feed-forward neural networks that have been pre-trained to mimic the original attention blocks' input-output behavior.

What problem does it solve and what is the "shape" of the solution? The paper doesn't solve a practical problem (the replacements are larger and less flexible than the original attention). Instead, it answers an analytical question: can attention be replaced by something simpler? The solution takes the shape of a component-by-component substitution experiment: train a vanilla Transformer as a teacher, extract intermediate input-output pairs from its attention blocks, train feed-forward networks to reproduce those mappings via supervised learning (knowledge distillation), then plug those trained networks back into the Transformer and measure how much translation quality degrades. By testing replacements at different levels of abstraction (just the attention computation, attention plus residual connection, individual attention heads, entire encoder layers) and for different attention types (encoder self-attention, decoder self-attention, decoder cross-attention), the paper maps out precisely where attention is replaceable and where it is not.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

1. Teacher Transformer — a standard 6-encoder, 6-decoder Transformer trained on IWSLT2017 translation data. This model achieves baseline BLEU scores (Table 2) and serves as the source of ground-truth intermediate activations for training the replacement networks.

2. Intermediate Activation Extractor — a data collection mechanism that runs the trained teacher Transformer on the training data and records input-output pairs at specific attention blocks. For example, for encoder self-attention replacement, it captures the input (concatenated word representations of a padded sentence) and output (updated word representations) of each of the six encoder self-attention layers.

3. Feed-Forward Replacement Networks — shallow one-hidden-layer neural networks of varying sizes (Table 1: from ~290K to ~46M parameters, compared to the original attention layer's ~60K parameters). Each network is trained independently via standard supervised learning to map the extracted input activations to the extracted output activations. Six independent networks are trained for each replacement approach (one per Transformer layer).

4. Architectural Variants — four structurally different ways of plugging the replacement network back into the Transformer, corresponding to different levels of abstraction (ALR, ALRR, ASLR, ELR; detailed in Section 3.4). Each variant determines what exactly the feed-forward network replaces and how its inputs are prepared and outputs are integrated.

5. Evaluation Pipeline — the modified Transformer, with trained replacement networks substituted in for the original attention blocks, is evaluated on the IWSLT2017 test set using the BLEU metric. BLEU scores are compared to the teacher Transformer's baseline to quantify how much performance is retained.

Information flows as follows: the teacher Transformer is trained normally → training sentences are passed through the teacher and intermediate activations are captured and preprocessed (padded to fixed length 50, flattened) → each replacement network is trained to map the input activation to the output activation at its assigned layer → the trained networks are inserted into a copy of the Transformer architecture, replacing the corresponding attention components → the modified Transformer translates test sentences → BLEU scores are computed and compared to baseline.

3.3 Roadmap for the Deep Dive

• First, the teacher Transformer setup and training, since all replacement networks depend on the quality and specific configuration of the teacher model from which they distill knowledge.
• Second, the data extraction and preprocessing pipeline, because the fixed-size nature of feed-forward networks requires significant data transformation that fundamentally constrains what the replacements can represent and how they operate.
• Third, the four architectural replacement variants (ALR, ALRR, ASLR, ELR) and their design rationales, since these define the different hypotheses being tested about what level of the attention mechanism's contribution is essential.
• Fourth, the specific feed-forward network architectures, their size variants, and the training procedure, including hyperparameters and the important distinction between encoder self-attention, decoder self-attention, and decoder cross-attention replacements.
• Fifth, how the trained networks are re-integrated into the Transformer for evaluation, since the different attention types require different input formatting and causality handling.
• Sixth, the design choices that distinguish this work: why knowledge distillation rather than end-to-end training, why fixed-size padding rather than dynamic architectures, and why component-by-component replacement rather than whole-model distillation.

3.4 Detailed, Sentence-Based Technical Breakdown

Teacher Transformer: The Source of Distilled Knowledge

The entire experimental pipeline begins with training a vanilla Transformer model following the architecture introduced by Vaswani et al. (2017). This model serves as the teacher in the knowledge distillation framework: its intermediate activations provide the supervised training targets for the replacement feed-forward networks.

Architecture details. The teacher Transformer consists of six encoder layers and six decoder layers. Each encoder layer contains a multi-head self-attention block followed by a position-wise feed-forward network, with residual connections and layer normalization around each sub-layer. Each decoder layer contains a masked multi-head self-attention block, a multi-head cross-attention block (attending to the encoder output), and a position-wise feed-forward network, again with residual connections and layer normalization.

Key modification from the original paper. The authors made one significant change to the standard Transformer configuration to manage computational demands:

"To reduce training times and make testing faster, we reduced the embedding size from 512 to 128."

The original Vaswani et al. Transformer uses a model dimension $d_{\text{model}} = 512$. Reducing this to 128 substantially decreases the number of parameters and the computational cost of both training the teacher and training the replacement networks. The authors note that these changes "did not drop the overall score too much below the original BLEU score but resulted in significantly lower computational power demands." This is an important contextual detail: the baseline against which replacements are compared is not the full-scale 512-dimensional Transformer from the original paper, but a smaller 128-dimensional variant trained on IWSLT2017.

Training data and procedure. The teacher Transformer is trained on the IWSLT2017 dataset across four language pairs: French-English (F2E), English-French (E2F), German-English (G2E), and English-German (E2G). Specific training hyperparameters for the teacher are not detailed in the paper beyond the embedding size modification, which is a notable omission—the reader cannot fully assess whether the teacher was optimally trained. The resulting baseline BLEU scores are presented in Table 2 of the paper:

These scores serve as the ceiling against which all replacement experiments are compared, with results reported as both absolute BLEU scores and relative percentages of the baseline.

The original attention layer parameter count. The paper states that the original attention layer has "60,000 parameters in our case." This number reflects the 128-dimensional embedding: a standard multi-head attention with $d_{\text{model}} = 128$ and presumably 8 heads (though the exact head count is not specified), giving each head a dimension of $128/8 = 16$. The total parameters come from the query, key, value, and output projection matrices: $4 \times (128 \times 128) = 65{,}536$, which rounds to approximately 60,000. This is the baseline against which the replacement network sizes in Table 1 should be compared.

Data Extraction and Preprocessing: Making Attention Look Like a Fixed-Dimension Mapping

The core technical challenge in replacing attention with feed-forward networks is that attention operates on variable-length sequences—the input is a set of $L$ word representations, where $L$ varies per sentence—while a standard feed-forward network requires a fixed-size input vector. The paper bridges this gap through a specific data transformation pipeline that converts variable-length sequences into fixed-size vectors via padding, with significant implications for both what the networks can learn and the flexibility of the resulting model.

Step 1: Extracting intermediate activations from the teacher. The trained teacher Transformer is run in inference mode on the training sentences. For each attention block targeted for replacement, the input and output activations are recorded. The input consists of word representations for all positions in the sentence before the attention operation; the output consists of the updated word representations after the attention operation (and, depending on the replacement variant, possibly after the residual connection and layer normalization as well). The authors describe this process:

"As the initial step, intermediate activations (input-output pairs) are extracted from the trained Transformer and used as training data for the desired Feed-Forward replacement network."

Figure 4 in the paper illustrates this extraction process for the ALRR variant, showing how the teacher Transformer's internal activations at a specific point in the encoder are captured and stored as (input, output) training pairs.

Step 2: Concatenation and padding to fixed size. The feed-forward network cannot accept a variable-length sequence of word vectors as input. The authors' solution is to treat the entire (padded) sentence as a single flat vector. Specifically:

"the FF network takes in the concatenated word representations of a sentence as input and produces updated word representations as output in a single pass."

If each word is represented by a 128-dimensional embedding, and sentences are padded to a maximum length of 50 words, then the input to the replacement network is a flat vector of dimension $50 \times 128 = 6{,}400$. The output is similarly structured: a flat vector of dimension 6,400 representing the updated word embeddings for all 50 positions. This means the feed-forward network must learn to process all positions of the sentence simultaneously in a single forward pass—it has no inherent notion of sequence or token order beyond what is implicitly encoded in the concatenation structure.

Step 3: Masking padded positions. Not all sentences are exactly 50 words long. For shorter sentences, the remaining positions are padded with zero vectors. To prevent the model from learning spurious dependencies on these zero-padded positions, the authors apply masking:

"we have decided to pad all sentences to a maximum fixed length and mask the padded values with zeros to prevent them from influencing the model's inference."

The masking ensures that the feed-forward network's computation on real token positions is not contaminated by processing zero-padded positions. However, the paper does not specify the exact masking mechanism—whether it is applied during training only, during inference only, or both—which is a minor ambiguity.

Step 4: Choosing the maximum sentence length. The choice of 50 as the maximum sequence length is data-driven:

"Since 96% of the samples in the datasets are of length 50 or less, the datasets were not significantly shrunk."

Sentences longer than 50 words are truncated or discarded (the paper does not specify which). The 96% coverage means that approximately 4% of the training data—those with sentences exceeding 50 words—is lost. This is an acceptable tradeoff for the analytical goals of the paper, but it represents a practical limitation: the replacement model fundamentally cannot process sequences longer than 50 tokens. The paper acknowledges this as a key disadvantage:

"Another downside of our replacement of the attention with a fixed-size feed-forward network is the imminent lack of flexibility of the model in terms of the length of sequences the model can operate with."

This contrasts sharply with the original Transformer, which can handle arbitrarily long sequences (limited only by memory and the quadratic cost of attention). The fixed-size constraint is a direct consequence of using feed-forward networks rather than a property of the replacement approach per se—one could imagine using recurrent or other sequence-length-agnostic architectures as replacements—but it represents a real practical limitation of the specific networks studied.

A crucial design implication. This data transformation fundamentally changes what the replacement network is being asked to learn. In the original Transformer, attention is position-equivariant: the computations applied at position $i$ are the same as those applied at position $j$, just with different inputs. The attention mechanism processes a sequence of arbitrary length with a fixed set of parameters by applying the same operation at each position. The feed-forward replacement, by contrast, learns a position-dependent mapping: the weights connecting input position 3 to output position 7 are different from those connecting input position 5 to output position 7, because the concatenated vector has a fixed spatial structure. This means the replacement network can, in principle, learn position-specific transformations—something the original attention cannot do—but it also means the network cannot generalize to sequence lengths it wasn't trained on, since position 51 would have no learned weights. This is a fundamental representational difference between the two approaches, not just an implementation detail.

The Four Replacement Variants: Ablating Attention at Different Levels

The paper proposes four structurally different approaches to replacing attention with feed-forward networks. Each variant operates at a different level of abstraction, replacing a different set of Transformer components and testing a different hypothesis about where attention's contribution lies.

Variant 1: Attention Layer Replacement (ALR)

ALR is the most conservative replacement: it replaces only the multi-head attention (MHA) computation itself, while preserving the surrounding architecture—the residual connection and the layer normalization—intact. As the paper states:

"ALR replaces only the multi-head attention (MHA) block with an FF network, keeping the residual connection and layer normalization intact."

What stays and what goes. The Transformer's encoder (or decoder) sub-layer consists of $\text{LayerNorm}(x + \text{MultiHeadAttention}(x))$. Under ALR, this becomes $\text{LayerNorm}(x + \text{FF}(x))$, where $\text{FF}$ is the trained feed-forward replacement network. The feed-forward network receives the same input the attention block would have received and must produce the output the attention block would have produced. The residual connection still adds the original input $x$ to this output, and layer normalization is still applied to the sum.

What this tests. ALR tests whether the computation performed by the attention mechanism—the query-key-value projections, the dot-product similarity scoring, the softmax normalization, and the weighted aggregation of values—can be replaced by a simple feed-forward mapping. Because the residual connection is preserved, the feed-forward network only needs to learn the residual that attention would have added to the input, which is potentially an easier learning problem than reproducing the full output. This is the most favorable setting for the replacement and, as the results show (Figure 2), the most successful approach.

Motivation for the residual connection's preservation. The authors deliberately designed ALR and ALRR (see below) as a pair:

"ALR and ALRR approaches were designed in such a way as to decouple the effects and benefits of the attention layer from the benefits brought by the residual connection."

By comparing ALR (residual preserved) to ALRR (residual removed), the experiment isolates whether any performance drop comes from failing to replicate attention's computation or from losing the residual pathway's optimization and representational benefits.

Variant 2: Attention Layer with Residual Connection Replacement (ALRR)

ALRR takes a more aggressive step: it replaces both the attention block and the residual connection with a single feed-forward network. The paper states:

"The MHA module, along with the residual connection is replaced by the FF network. This approach effectively removes the residual connection when the FF Network is substituted in the Transformer."

What this means structurally. The Transformer sub-layer $\text{LayerNorm}(x + \text{MultiHeadAttention}(x))$ becomes $\text{LayerNorm}(\text{FF}(x))$. The feed-forward network now must produce not just the attention output, but $x + \text{MultiHeadAttention}(x)$—the sum of the input and the attention output—before layer normalization. This is a harder learning problem because the network must internally decide how much of the original input to preserve and how much to transform, without the explicit "copy the input" shortcut that a residual connection provides.

What this tests. ALRR tests whether the entire sub-layer computation, including the beneficial optimization properties of residual connections, can be compressed into a single feed-forward mapping. The residual connection is known to be critical for training deep Transformers (it allows gradients to flow directly through the network without attenuation, mitigating vanishing gradients). If a feed-forward network can learn to replicate the input-plus-attention function well enough to achieve competitive BLEU scores, it suggests that the residual connection's contribution is, at least for a trained network, representable as a learned computation rather than requiring an explicit architectural pathway.

Performance implications. As the results in Figure 2 and Table 3 show, ALRR is viable but requires larger networks to work: the XS and S sizes collapse to near-zero BLEU (e.g., ALRR Enc SA XS achieves 0.013 on E2G vs. 0.257 baseline), while the L size recovers to within a few BLEU points of the baseline (0.243 on E2G). This suggests that the residual connection provides a strong inductive bias that makes the function easier to learn, and that reproducing that function without the architectural shortcut requires substantially more capacity.

Variant 3: Attention Separate Heads Layer Replacement (ASLR)

ASLR introduces a structural mimicry of multi-head attention's internal organization. Instead of replacing the entire multi-head attention block with one feed-forward network (as in ALR), ASLR replaces each individual attention head with a separate, smaller feed-forward network. The paper explains:

"As a variant of the ALR, this method replaces every single head of the MHA module with a separate FF network."

Structural analogy to multi-head attention. In standard multi-head attention, the input is projected into $h$ separate query, key, and value spaces (one per head), attention is computed independently in each head, and the head outputs are concatenated and projected back to the model dimension. In ASLR, each head becomes its own feed-forward network that receives the full input (not a projected subspace) and produces a head-specific output. The outputs of all head-specific networks are then concatenated and projected—presumably through a learned aggregation—to produce the final output of the replacement block.

What this tests. ASLR tests whether the multi-head decomposition—the idea that attention benefits from computing multiple independent similarity patterns in parallel and then integrating them—is important for the function being learned. If a single monolithic feed-forward network (ALR) can match the performance, the multi-head structure is not necessary for representational purposes. If ASLR outperforms ALR at comparable total parameter counts, the head-wise decomposition provides a useful inductive bias.

Parameter count note. The paper's Table 1 reports ASLR parameters "on a per-attention-head basis." For example, ASLR size S has 1.5M parameters per head. If there are 8 heads (a reasonable assumption given $d_{\text{model}} = 128$), the total ASLR S replacement would have $8 \times 1.5\text{M} = 12\text{M}$ parameters per layer, making it substantially larger than the corresponding ALR S (640K) or even ALR M (10M). This is an important comparison nuance: ASLR's success or failure must be interpreted in light of its much larger total parameter budget.

Results context. From Table 3 and Figure 2, ASLR performs competitively with ALR at larger sizes, but the performance gains relative to ALR are modest. This suggests that the multi-head decomposition, while not harmful, is not the primary source of attention's effectiveness for this task—a single monolithic network can learn to replicate the behavior without explicit head-wise parallelism.

Variant 4: Encoder Layer Replacement (ELR)

ELR is the most aggressive replacement: it replaces the entire encoder layer—including the self-attention block, both residual connections, both layer normalizations, and the position-wise feed-forward network—with a single feed-forward network. The paper states:

"As the highest level of abstraction, the whole encoder block is replaced with an FF Network in the ELR method. This essentially upends the original encoder architecture, turning it into a sequence of FF networks - one for each block of the encoder."

What this means structurally. A standard encoder layer computes:

$\text{out} = \text{LayerNorm}(\text{FF}(\text{LayerNorm}(x + \text{Attention}(x))) + \text{LayerNorm}(x + \text{Attention}(x)))$

Under ELR, this entire computation is replaced by a single function:

$\text{out} = \text{FF}_{\text{ELR}}(x)$

The feed-forward network must learn to perform the equivalent of attention, residual combination, normalization, and feed-forward transformation all in one monolithic mapping. There are no residual connections, no normalization layers, no sub-structure of any kind—just a sequence of independent feed-forward networks, one per original encoder layer.

What this tests. ELR tests the most extreme hypothesis: is any of the Transformer's architectural structure necessary, or can the entire computation performed by an encoder layer be captured by a sufficiently expressive generic function approximator? A positive result here would suggest that the Transformer's layered, attention-based architecture is entirely an optimization convenience rather than a representational necessity—that the same function could be learned by a stack of simple feed-forward networks if only we knew how to optimize them.

Performance implications. Unsurprisingly, ELR performs the worst of all four variants (Figure 2). Even at the largest size (L), ELR BLEU scores (e.g., 0.194 on E2G) fall substantially short of the baseline (0.257) and of the other replacement variants. The paper attributes this to "the simplicity of the replacement model, which discards all of the encoder structures that aid training." This is the key insight: the Transformer's architectural components (residual connections, layer normalization, attention's specific computational structure) are not strictly necessary for representing the function, but they are critical for making that function learnable through gradient-based optimization. Without them, even a high-capacity feed-forward network trained with distillation cannot fully recover the teacher's behavior.

Summary of the four variants' design rationale. These four variants are not an arbitrary collection of architectural tweaks. They form a systematic ablation hierarchy, each removing one more structural element of the Transformer:

1. ALR: Remove only the attention computation; keep residuals and norms.
2. ALRR: Remove attention computation + residual connection; keep only layer norm.
3. ASLR: Same level as ALR but tests the importance of multi-head decomposition.
4. ELR: Remove everything; test whether any Transformer-specific structure is needed.

This hierarchy allows the paper to localize precisely which structural elements matter: if ALR works but ALRR doesn't, residuals matter; if ALRR works but ELR doesn't, layer normalization or the feed-forward sub-layer matters; if ASLR outperforms ALR, multi-head structure matters. The results (Figure 2) show a clear gradient: performance degrades as more structural elements are removed, suggesting that each component—attention, residuals, layer normalization, feed-forward sub-layer—contributes incrementally to the representational or optimization quality of the teacher's function.

Feed-Forward Network Architecture and Training

The replacement networks share a common simple architecture across all variants, differing only in their input/output dimensions and hidden layer size. The networks are trained through standard supervised learning with the teacher Transformer's intermediate activations as targets.

Network architecture: shallow and simple. Every replacement network is a one-hidden-layer feed-forward neural network. The paper is explicit about this simplicity:

"As a replacement module in all of these cases, the simple, shallow one-hidden-layer FF network was used."

This is a deliberate design choice, not an oversight. The research question is whether attention can be replaced by the simplest possible alternative architecture. Using deep, complex networks with skip connections or attention-like mechanisms of their own would defeat the analytical purpose: the goal is to test the extreme case. If a single-hidden-layer network—essentially a universal function approximator with no architectural priors for sequence processing—can match attention's behavior, then attention's architectural specificity is not providing any representational advantage for the learned function.

Size variants and parameter scaling. The paper trains networks at five different sizes for each replacement variant, labeled XS through L. The exact parameter counts for a single replacement network are given in Table 1:

The paper notes that ALRR and ELR use "the same number of parameters" as ALR (stated in the appendix: "ALR, ALRR, and ELR share the same number of parameters"), so the ALR row represents all three. ASLR's numbers are per-head, so total parameters scale with the number of heads.

Scale relative to the original attention layer. A critical comparison: the original attention layer has approximately 60,000 parameters. The smallest replacement network (ALR XS at 320K) is already more than 5× larger; the largest (ALR L at 41M) is nearly 700× larger. The paper acknowledges this explicitly: "The original number of parameters for the attention layer (60,000 parameters in our case) is mostly exceeded by the replacement networks." This is a central limitation of the approach: the feed-forward networks are not a more efficient replacement for attention; they achieve comparable performance only through a massive increase in parameter count. This is attributed to the fixed-size input processing:

"mainly due to fixed-size inputs and outputs, and the processing format demanded by the FF network."

The fixed-size concatenated input (6,400 dimensions for a 50-word sentence with 128-dimensional embeddings) creates an enormous input space, and the first layer of the feed-forward network must map from this 6,400-dimensional input to the hidden layer. This first-layer weight matrix alone contributes the bulk of the parameter count: a hidden layer of size $H$ requires $6{,}400 \times H + H \times 6{,}400$ parameters just for the input-to-hidden and hidden-to-output matrices (since the output must also be 6,400 dimensions to reconstruct the updated word representations). For ALR XS, this implies a hidden layer size on the order of $320{,}000 / (2 \times 6{,}400) \approx 25$ neurons, while ALR L would have approximately $41{,}000{,}000 / (2 \times 6{,}400) \approx 3{,}200$ neurons.

Training procedure. All replacement networks are trained with the same configuration:

• Optimizer: Adam
• Learning rate: 0.001
• Batch size: 1400
• Epochs: 20
• Training data: Input-output activation pairs extracted from the trained teacher Transformer on the IWSLT2017 training set

The authors state: "Training settings were kept the same for all networks." This uniformity ensures fair comparison across variants and sizes, though it means the hyperparameters are not individually tuned per configuration—suboptimal hyperparameters for larger or smaller networks could influence the results. The paper's future work section acknowledges this: "further optimization of the FF networks' hyperparameters using advanced parameter search (e.g. using Bayesian optimization (Snoek, Larochelle, and Adams 2012)) could yield even better results."

Number of networks trained. For each replacement variant and each size, six independent networks must be trained—one for each of the six encoder (or decoder) layers. This is because each layer's attention block performs a different function (lower layers might learn local syntactic patterns, higher layers more abstract semantic relationships), so the input-output mapping differs per layer. The paper states: "Every approach and every FF network size demanded training 6 independent networks for each of the self-attention or the cross-attention blocks." For a full sweep across all four variants, five sizes, and six layers, this amounts to $4 \times 5 \times 6 = 120$ trained networks for encoder replacement alone, though the paper does not specify which experiments were run exhaustively.

Loss function. The paper does not explicitly state the loss function used for training the feed-forward networks, which is a notable omission. Given that the target is continuous activation vectors (the teacher's output word representations), the natural choice would be mean squared error (MSE) between the predicted and target activation vectors. The paper's reference to knowledge distillation (Hinton, Vinyals, and Dean 2015) suggests the possibility of using soft targets with temperature scaling, but the original distillation paper's approach of matching output probabilities doesn't directly apply to intermediate activation matching. The most plausible interpretation is that the networks are trained with simple MSE regression: minimize $\|\text{FF}(x) - y_{\text{teacher}}\|^2$ where $x$ is the input activation and $y_{\text{teacher}}$ is the teacher's output activation at that layer.

Decoder-Specific Considerations: Causality and Cross-Attention

The decoder introduces two complications not present in the encoder: the self-attention is masked (causal), and there is an additional cross-attention block that attends to the encoder's output. These differences require modifications to the data preparation and network structure.

Decoder self-attention replacement: handling causality. In the decoder's masked self-attention, each position can only attend to itself and previous positions—not to future positions. This is essential for autoregressive generation, where the model predicts one token at a time and must not "cheat" by looking ahead. The feed-forward replacement must respect this causality constraint. The paper's solution:

"The network processes each word individually by feeding the entire sentence representation through the network and masking the representation of words that come after the word is processed. This is done to take into account the concept of causality, where only the previous words in the sentence can affect the meaning of the current word."

Unlike the encoder replacement, where the entire padded sentence is passed through the network at once to produce all updated representations simultaneously, the decoder self-attention replacement processes the sentence one position at a time. For position $i$, the full padded sentence representation is fed as input, but positions $i+1$ through 50 are masked to zero. The network then produces the updated representation for position $i$. This process is repeated for each position, making decoder self-attention replacement $L$ times more computationally expensive at inference time than encoder replacement (where $L$ is the sequence length, up to 50).

An important implication: the decoder self-attention replacement network has the same architecture (same input and output dimensions) as the encoder replacement network but is used differently—called repeatedly with different masks rather than once with the full unmasked input. The paper does not clarify whether the same trained network handles both masked and unmasked regimes (which would require learning to respect the mask), or whether separate networks are trained for each regime. The training data presumably consists of teacher activations from the masked self-attention block, so the network learns the masked behavior from the teacher, but the inference-time procedure of applying the network position-by-position with explicit masking is described as a runtime mechanism.

Decoder cross-attention replacement: processing two input sequences. Cross-attention has a fundamentally different input structure from self-attention:

"Cross-attention in the decoder accepts both word representations from the encoder and decoder layers, integrating them together."

The cross-attention block receives two sets of representations: the decoder's self-attention output (serving as queries) and the encoder's output (serving as keys and values). The attention mechanism computes compatibility between decoder positions and encoder positions, allowing the decoder to selectively attend to relevant parts of the source sentence.

To replicate this with a feed-forward network:

"word representations from both encoder and decoder were concatenated together and padded, having the input size doubled in comparison to the self-attention replacement networks."

The input to the cross-attention replacement network is the concatenation of the padded encoder output (50 positions × 128 dimensions = 6,400 features) and the padded decoder self-attention output (another 50 × 128 = 6,400 features), yielding a total input dimension of 12,800. The output remains 6,400 dimensions (the updated decoder representations). This doubling of the input dimension further inflates the already-large parameter count of the replacement networks, since the first-layer weight matrix must now be $12{,}800 \times H$ rather than $6{,}400 \times H$.

Why cross-attention replacement fails. The results in Figure 3 and Table 3 show a dramatic performance collapse when cross-attention is replaced: at the largest size (L), cross-attention-only replacement achieves BLEU scores of 0.115–0.130 across language pairs, compared to baseline scores of 0.257–0.324. Even when only cross-attention is replaced (with self-attention left intact), performance drops to roughly 35–40% of baseline. The paper's hypothesis:

"This suggests that the proposed shallow networks were not able to capture the more intricate and complex interactions between the differing sequences that enter the cross-attention mechanism."

The cross-attention's function may be genuinely more complex than self-attention's: it must learn to align two sequences in different languages, matching source words to their target translations, handling reordering, and managing the many-to-many alignment patterns characteristic of machine translation. A shallow feed-forward network, even with 12,800-dimensional input, may lack the inductive bias or capacity to learn this alignment implicitly from fixed-size concatenated representations. This is arguably the paper's most important negative result: it establishes a clear boundary between replaceable and non-replaceable attention, showing that attention's cross-sequence alignment capability is harder to replicate than its within-sequence contextualization capability.

Re-Integration and Evaluation: Putting the "Attentionless Transformer" to the Test

After all replacement networks are trained, they must be inserted back into the Transformer architecture and evaluated on translation quality.

Insertion procedure. The trained feed-forward networks replace the corresponding attention components in a copy of the original Transformer architecture. For ALR, this means the $\text{MultiHeadAttention}(x)$ call is replaced by $\text{FF}_{\text{ALR}}(x)$, but the surrounding structure (residual addition, layer normalization) is preserved. For ALRR, $x + \text{MultiHeadAttention}(x)$ is replaced by $\text{FF}_{\text{ALRR}}(x)$. For ELR, the entire encoder layer function is replaced by $\text{FF}_{\text{ELR}}(x)$. This is illustrated in Figure 4 for the ALRR approach: the original encoder layer on the left, with its attention block, is modified to the architecture on the right, where the attention block (or attention plus residual, depending on the variant) is replaced by a trained FF network.

Inference procedure. During translation, the modified Transformer operates almost identically to the original. For encoder self-attention replacement, each encoder layer processes the padded input sequence by passing the concatenated word representations through the replacement FF network (instead of computing attention), adding the residual connection (if preserved), and applying layer normalization. The decoder operates normally (unless its attention blocks are also replaced). For full replacement (encoder and decoder self-attention and cross-attention all replaced), the entire forward pass uses only feed-forward networks, residual connections, and layer normalization—no dot-product attention computations whatsoever.

Evaluation metric. Performance is measured using the BLEU score (Papineni et al., 2002), the standard metric for machine translation quality. BLEU measures the n-gram overlap between the model's output and one or more reference translations, producing a score between 0 and 1 (the paper reports scores as decimals rather than percentages, so a BLEU of 0.257 means 25.7% n-gram match with the reference). Higher is better. The paper reports both absolute BLEU scores (Table 3) and relative BLEU scores as percentages of the baseline Transformer's score (Figures 2 and 3).

Experimental configurations. The paper tests three progressively more aggressive replacement scenarios:

1. Encoder self-attention only: Only the six encoder self-attention blocks are replaced. The decoder remains the original Transformer decoder. This tests whether feed-forward networks can learn the bidirectional contextualization that encoder self-attention provides. All four replacement variants (ALR, ALRR, ASLR, ELR) are tested in this configuration.

2. Encoder and decoder self-attention (E-D SA): Both encoder and decoder self-attention are replaced, but the decoder cross-attention remains intact. This tests whether feed-forward networks can handle both bidirectional (encoder) and causal (decoder) self-attention. Only ALR is tested in this configuration (full results in Table 3, rows labeled "E-D SA").

3. Full replacement: All attention blocks—encoder self-attention, decoder self-attention, and decoder cross-attention—are replaced with feed-forward networks. This is the "attentionless Transformer." Only ALR is tested in this configuration (Table 3, rows labeled "Full").

Additionally, decoder cross-attention only and decoder self-attention only replacements are tested in isolation (Table 3, rows labeled "Dec CA" and "Dec SA"), allowing the contribution of each attention type to be assessed independently.

Design Choices: Why Knowledge Distillation Instead of End-to-End Training?

The most consequential methodological choice in this paper is the use of knowledge distillation rather than end-to-end training. The replacement networks are not trained as part of the Transformer with a translation objective; they are trained separately to mimic the teacher's intermediate activations. This choice has profound implications for what the results mean.

What distillation enables that end-to-end training would not. If the authors had trained the replacement networks end-to-end as part of the Transformer (by backpropagating the translation loss through the feed-forward networks), the experiment would answer a different question: "Can feed-forward networks be optimized via gradient descent to perform a function that supports translation?" A negative result (poor BLEU) could mean either that feed-forward networks cannot represent the necessary function, or that gradient descent cannot find good parameters for them in this context. The paper's central claim—that attention is replaceable—requires separating these two explanations.

Knowledge distillation separates them. By using the teacher's activations as supervised targets, the feed-forward networks are trained with a direct, well-behaved regression objective (matching specific activation vectors) rather than the complex, high-variance translation loss that depends on the entire model's behavior. If the networks can achieve low regression error on the activation-matching task, but end-to-end training would fail, the bottleneck is optimization, not representation. The paper's concluding statement reflects exactly this interpretation:

"These conclusions also point out the deficiencies of the current optimization methods, which are not able to train these 'attentionless Transformers' from scratch but need more advanced techniques, such as knowledge distillation to converge into desired parameter configurations."

The optimization hypothesis. The implication is that attention provides a critical inductive bias for optimization—its specific computational structure (dot-product similarity, softmax normalization, weighted aggregation) creates a loss landscape where gradient descent reliably finds good parameters. The feed-forward networks, though formally capable of representing the same function (as evidenced by their ability to learn it through distillation), do not provide the same favorable optimization landscape. From a random initialization, gradient descent on the translation loss gets stuck in poor local minima or saddle points; the distillation objective, being a simpler regression problem with direct supervision, provides a clearer path to good parameters.

What this means for interpreting the results. The paper's demonstration that "attentionless Transformers" can match the original Transformer's performance should not be interpreted as "you can train Transformers without attention." It should be interpreted as "attention is not the only architecture that can represent the computations needed for translation, but it may be the only architecture we currently know how to optimize effectively for this task." This is a more nuanced claim that separates representational necessity from optimization convenience—a distinction that is crucial for understanding what architectural innovations actually contribute.

Connection to the broader deep learning narrative. This finding parallels results in other domains. The lottery ticket hypothesis (Frankle and Carbin, 2019) shows that sparse subnetworks exist within randomly initialized networks that can train to full accuracy, but finding them requires special procedures. The fact that attention can be replaced by feed-forward networks when trained with distillation suggests a similar phenomenon: good solutions exist in the parameter space, but standard optimization cannot discover them without the right architectural scaffolding. The transformer's attention mechanism provides that scaffolding.

4. Key Insights and Innovations

Innovation 1: Attention Is an Optimization Scaffold, Not a Representational Necessity

The paper's deepest conceptual contribution is not that feed-forward networks can replace attention (Section 3 documents the mechanism), but what that replacement reveals about why attention matters in the first place. The dominant narrative since Vaswani et al. (2017) has been that attention's dynamic, content-dependent routing is computationally essential for modeling long-range dependencies—that the ability to compute pairwise similarity between all positions and aggregate accordingly provides a representational capability that simpler architectures lack. This paper challenges that narrative at its foundation by demonstrating that a static, one-hidden-layer feed-forward network—the most generic function approximator available, with no built-in notion of sequences, similarity, or dynamic weighting—can reproduce attention's input-output mapping closely enough to maintain translation quality within 1–2 BLEU points of the original (ALR L: 0.252 vs. 0.257 baseline on English-German; Table 3).

What makes this finding intellectually distinctive is the separation it enforces between representation and optimization. Prior work on model distillation (Ba and Caruana, 2014; Urban et al., 2017) demonstrated that deep convolutional networks could be compressed into shallower architectures, but the target there was the entire model's output, and the replacements were typically architecturally related to the originals (shallower but still convolutional). This paper does something different: it isolates a specific computational primitive (multi-head attention), replaces it with something architecturally unrelated (a shallow MLP), and uses the success of the replacement to diagnose what attention actually contributes. The diagnosis is striking:

"These conclusions also point out the deficiencies of the current optimization methods, which are not able to train these 'attentionless Transformers' from scratch but need more advanced techniques, such as knowledge distillation to converge into desired parameter configurations."

The logic chain here is subtle but important. If feed-forward networks can represent the function (proven by successful distillation) but cannot discover it through end-to-end training (implicit in the need for distillation), then attention's contribution is not representational but optimizational. Attention provides a loss landscape where gradient descent reliably finds good solutions. The dot-product similarity computation, the softmax normalization, the weighted aggregation—these specific operations create gradients that flow in directions that lead to useful representations. A generic feed-forward network has the capacity to represent the same mapping but lacks the inductive bias that makes that mapping discoverable from random initialization with a translation loss.

This reframing has implications beyond this specific paper. It suggests that much of neural architecture design—the careful crafting of attention mechanisms, convolutions, recurrent cells—may be better understood as optimization engineering rather than representation engineering. The architectures work not because they are the only forms capable of expressing the target function, but because they are forms that gradient-based optimization can effectively shape. This perspective aligns with the lottery ticket hypothesis (Frankle and Carbin, 2019) and related work showing that successful architectures can be viewed as providing "winning tickets" that standard optimization can find. The paper's evidence for this claim is indirect but coherent: the existence of feed-forward parameters that match attention's behavior (via distillation) proves representation is not the bottleneck, and the acknowledged failure of end-to-end training for these same architectures points squarely at optimization.

A boundary condition sharpens this insight: the optimization story applies specifically to self-attention, not cross-attention. Cross-attention replacement fails dramatically (Table 3: Dec CA L achieves 0.115–0.130 BLEU vs. 0.257–0.324 baseline), suggesting that cross-attention's function involves genuinely more complex computations—aligning representations across different languages—that may exceed what these shallow networks can represent at the tested scales, not just what they can discover through optimization. This asymmetry is consistent with the optimization-scaffold hypothesis: self-attention's within-sequence contextualization may be a simpler function that distillation can extract but optimization cannot find, while cross-attention's cross-sequence alignment may be genuinely harder to represent.

Innovation 2: Component-Level Ablation Through Surgical Replacement

The paper's second distinctive contribution is methodological: the component-by-component replacement paradigm as a diagnostic tool for understanding neural architectures. Prior work on replacing complex models with simpler ones (Ba and Caruana, 2014; Urban et al., 2017; Hinton et al., 2015) operated at the granularity of entire models—compress the whole teacher into a smaller student. This paper operates at the granularity of individual sub-layers, replacing exactly the attention computation while preserving surrounding architecture (ALR), or attention plus residual (ALRR), or entire encoder layers (ELR), or individual attention heads (ASLR). This surgical approach transforms the replacement from a compression technique into an ablation instrument.

The intellectual move is analogous to lesion studies in neuroscience: by selectively removing or replacing specific components and observing the effect on behavior, one can infer the functional role of each component. The four replacement variants form an ablation hierarchy (as detailed in Section 3.4) that allows the paper to localize the contribution of each Transformer structural element:

• ALR removes only the attention computation while preserving residuals and layer normalization → if this works, the specific dot-product/softmax mechanism of attention is replaceable.
• ALRR additionally removes the residual connection → the performance gap between ALR and ALRR isolates the residual pathway's contribution. The fact that ALRR requires much larger networks to recover (Table 3: ALRR Enc SA XS collapses to 0.013 on E2G, while ALRR L reaches 0.243) quantifies the residual connection's role as a learnability aid.
• ELR removes everything—attention, residuals, normalization, feed-forward sub-layer → the gap between this and ALRR isolates the contribution of layer normalization and the sub-layer structure itself. ELR L reaches only 0.194 on E2G versus ALRR L's 0.243, showing that even when the attention computation and residual are removed, the remaining Transformer scaffolding (layer norm, the position-wise FF sub-layer) still provides meaningful structure.
• ASLR replaces individual attention heads rather than the whole block → comparison with ALR isolates whether the multi-head decomposition matters. The modest performance difference (ASLR tracks ALR closely; Table 3) suggests it is not the primary source of effectiveness for this task.

What distinguishes this from standard ablation studies is the use of distillation rather than zeroing or randomization as the removal method. A typical ablation would remove attention by setting its outputs to zero or replacing them with random noise—both of which destroy information and test only whether the component is necessary, not whether something simpler could suffice. By training a replacement to mimic the removed component, the paper tests substitutability: can a simpler architecture fill the same functional role? This is a more nuanced question than necessity, and the answer varies across components and attention types—self-attention is substitutable, cross-attention is not, residuals help but are not absolutely required, the encoder's entire structure provides incremental rather than binary value.

The practical value of this methodology extends beyond this paper's findings. The component-replacement paradigm provides a template for investigating other architectural questions: Can convolutional layers in vision models be replaced by simpler operations? Can the gating mechanisms in LSTMs be simplified? The key is the combination of targeted replacement (not whole-model compression), knowledge distillation (to factor out optimization effects), and graduated abstraction (replacing at different levels to localize effects). Any future work that asks "does architecture X actually need component Y?" can adopt this template.

Innovation 3: The Self-Attention/Cross-Attention Asymmetry as a Diagnostic Boundary

The paper's third distinctive contribution is demonstrating—for the first time, to the extent of these authors' investigation—that not all attention is equivalently replaceable, and that this asymmetry reveals a functional hierarchy within the Transformer's attention mechanisms. Prior work treated attention as a monolithic operation: multi-head scaled dot-product attention, applied identically (with masking variations) in all three Transformer attention blocks. The paper's decision to test encoder self-attention, decoder self-attention, and decoder cross-attention independently (Table 3: separate rows for Enc SA, Dec SA, Dec CA) reveals a sharp performance cliff that would be invisible in a whole-model replacement study.

The numbers tell a clear story. Encoder self-attention replacement with ALR L achieves BLEU scores nearly matching the baseline across all language pairs (e.g., 0.252 vs. 0.257 on E2G; 0.327 vs. 0.324 on G2E). Decoder self-attention replacement similarly succeeds (Dec SA L: 0.253 on E2G, 0.323 on G2E). But decoder cross-attention replacement collapses performance even at the largest network size: Dec CA L achieves only 0.115 on E2G, 0.130 on G2E—roughly 35–40% of the baseline. When all three are replaced (Full L), the cross-attention bottleneck dominates: 0.105 on E2G, 0.134 on G2E.

This asymmetry is theoretically informative. Self-attention—whether bidirectional in the encoder or causal in the decoder—operates on a single sequence, learning contextualized representations where each position incorporates information from other positions in the same sentence. Cross-attention operates on two distinct sequences in different languages, learning to align source representations with target representations. The failure of feed-forward networks to replicate cross-attention suggests that this cross-lingual alignment involves computations that are fundamentally harder to capture with a static, fixed-size mapping. The paper's hypothesis—that "the proposed shallow networks were not able to capture the more intricate and complex interactions between the differing sequences"—is somewhat vague, but the empirical boundary is crisp.

The significance of this finding extends beyond the paper's immediate scope. It suggests that attention's value is not uniform across its applications and that different attention mechanisms in a single architecture serve qualitatively different roles. Encoder self-attention contextualizes within a language—a function that, at least for the IWSLT2017 dataset and the 128-dimensional Transformer tested, can be captured by position-dependent feed-forward mappings. Cross-attention performs cross-lingual alignment—a function that resists such simplification. This distinction has implications for architecture design: if self-attention is replaceable but cross-attention is not, a hybrid architecture that uses feed-forward replacements for encoder layers while retaining cross-attention could reduce computational cost without sacrificing quality. It also suggests that future work on attention replacement should treat different attention types as distinct targets rather than assuming uniform applicability.

Innovation 4: The Parameter-Capacity Tradeoff as a Lens on Architectural Efficiency

The paper's fourth contribution is less a claimed innovation and more an emergent diagnostic from its experimental design: the explicit quantification of how many more parameters a generic architecture needs to match a specialized one. The replacement networks are not more efficient than attention—they are dramatically less efficient. The original attention layer has ~60K parameters; the ALR replacement networks range from 320K (XS) to 41M (L), representing a 5× to 700× inflation (Table 1). Even the best-performing ALR L, which nearly matches baseline BLEU, requires roughly 680× more parameters than the attention layer it replaces.

This parameter inflation is not presented as a failure but as an informative signal. It quantifies the architectural efficiency premium that attention provides: attention achieves a given level of sequence-processing performance with far fewer parameters than a generic feed-forward network because its structure (parameter sharing across positions, the dot-product similarity kernel) provides an inductive bias that is well-matched to the task. The feed-forward replacement must learn position-specific transformations (since the concatenated input has fixed spatial structure) and has no built-in notion of similarity or sequence, so it requires many more parameters to approximate the same function.

This finding connects to broader discussions about the role of architectural priors. The efficiency gap—680× parameters to match attention's performance—can be interpreted as a measure of how much the attention mechanism "knows" about sequence processing tasks a priori. Convolutions encode translational invariance; attention encodes a form of relational reasoning via pairwise comparison; feed-forward networks encode nothing about sequence structure at all. The parameter inflation needed to compensate for the lack of priors is a concrete, quantitative measure of how valuable those priors are. This is a conceptual contribution to the architecture design literature: it provides a methodology for measuring the "prior value" of an architectural component by measuring the capacity cost of replacing it with a prior-free alternative.

The qualification is important: this is a lower bound on the efficiency gap because the replacement networks also benefit from the teacher Transformer's learned representations. An attentionless Transformer trained end-to-end from scratch would likely require even more parameters to match Transformer performance than the distilled version, because optimization would be harder (as discussed in Innovation 1). So the 680× figure represents the minimal parameter overhead when the representation problem is solved (via distillation) but the architectural prior is absent. The true cost of replacing attention—including optimization difficulties—would be higher still.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the IWSLT2017 dataset (Cettolo et al., 2017), a standard benchmark for spoken language translation. The paper uses four language pairs: French-English (F2E), English-French (E2F), German-English (G2E), and English-German (E2G). After applying the maximum sentence length constraint of 50 words (which eliminates approximately 4% of samples), each subset contains roughly 200,000 training sentences and 1,000 test sentences. The authors state: "Since 96% of the samples in the datasets are of length 50 or less, the datasets were not significantly shrunk."

• Base model (teacher). The teacher is a standard Transformer following Vaswani et al. (2017) with six encoder layers and six decoder layers. The one modification from the original architecture is a reduction in embedding size: "To reduce training times and make testing faster, we reduced the embedding size from 512 to 128." This creates a substantially smaller model than the original 512-dimensional Transformer, with the multi-head attention layer containing approximately 60,000 parameters. The teacher is trained on each of the four language pairs independently, producing the baseline BLEU scores reported in Table 2: 0.257 (E2G), 0.324 (G2E), 0.276 (E2F), and 0.292 (F2E). The paper notes that this dimensionality reduction "did not drop the overall score too much below the original BLEU score" but provides no comparison to a 512-dimensional model trained on the same data to quantify the performance impact.

• Metrics. Translation quality is measured exclusively using the BLEU score (Papineni et al., 2002), reported as a decimal between 0 and 1 (where 0.257 represents 25.7% n-gram overlap with reference translations). The paper presents both absolute BLEU scores (Table 3) and relative BLEU scores, computed as a percentage of the teacher Transformer's baseline score on each dataset (Figures 2 and 3). The relative scores are averaged across the four language pairs to produce aggregate metrics, which the paper uses for its main summary figures. No other evaluation metrics (e.g., chrF, TER, human evaluation) are reported.

• Baselines. The sole baseline is the teacher Transformer's BLEU score on each language pair (Table 2). All replacement experiments are compared against this teacher, with performance reported as both absolute BLEU and the percentage of teacher BLEU retained. There is no comparison to alternative sequence models (LSTMs, CNNs), to the original 512-dimensional Transformer, or to other knowledge distillation approaches. The baseline is not augmented with any test-time strategies (beam search, ensembling) beyond standard greedy or beam decoding, though the exact decoding procedure is not specified.

• Generation budget / compute accounting. The paper does not measure or report computational cost in FLOPs, training time, or inference latency. The primary cost metric is parameter count: the number of trainable parameters in each replacement network compared to the original attention layer's ~60,000 parameters. This is reported in Table 1 across four size categories (XS through L) ranging from 290K to 46M parameters per network. The paper acknowledges that the replacements substantially exceed the original attention layer in parameter count: "The original number of parameters for the attention layer (60,000 parameters in our case) is mostly exceeded by the replacement networks, mainly due to fixed-size inputs and outputs, and the processing format demanded by the FF network." No compute-matched comparison (e.g., limiting the replacement network to the same parameter count or FLOPs as the attention layer) is attempted. The training cost is uniform across experiments—20 epochs with batch size 1400, learning rate 0.001, Adam optimizer—but total training FLOPs are not reported.

• Cross-validation / statistical protocol. The paper reports no cross-validation, no statistical significance testing, and no confidence intervals on BLEU scores. Results are reported as point estimates from a single evaluation run on the IWSLT2017 test set. The paper does not specify whether test-set performance was used for model selection (which would invalidate the reported scores as unbiased estimates of generalization) or whether a separate validation set was used for development. No ablation of random seeds or training run variance is provided.

Main Quantitative Results

The paper organizes its experiments around two axes of investigation: (1) encoder self-attention replacement across all four replacement variants and sizes, establishing which approaches can match the teacher's performance; and (2) full Transformer attention replacement, testing whether the best encoder approach generalizes to decoder self-attention and cross-attention, with results broken out by attention type.

Encoder Self-Attention Replacement: ALR and ASLR Match Baseline at Largest Sizes

The headline result for encoder-only replacement is presented in Figure 2 (Relative BLEU scores [%]) and detailed numerically in Table 3. At the largest network size (L), two of the four proposed methods achieve BLEU scores within approximately 1–3 percentage points of the teacher baseline:

• ALR L (Attention Layer Replacement): Achieves absolute BLEU scores of 0.252 (E2G), 0.327 (G2E), 0.276 (E2F), and 0.288 (F2E). The relative BLEU scores are approximately 98% (E2G relative to 0.257 baseline), 100.9% (G2E relative to 0.324—slightly exceeding the teacher), 100% (E2F), and 98.6% (F2E). Averaged across all four datasets, ALR L retains close to 99% of the teacher's translation quality. This is the paper's strongest positive result, demonstrating that a shallow feed-forward network can almost perfectly replicate the function of the encoder's multi-head self-attention when the residual connection and layer normalization are preserved.

• ASLR L (Attention Separate Heads Layer Replacement): Achieves scores of 0.252 (E2G), 0.326 (G2E), 0.271 (E2F), and 0.290 (F2E). Performance tracks ALR L very closely, with relative scores within 1–2% of ALR across all language pairs. The near-identical performance suggests that the multi-head decomposition—replacing each attention head with a separate feed-forward network rather than using one monolithic network—provides no meaningful advantage for this task at this scale.

The performance of ALRR and ELR, by contrast, falls substantially short:

• ALRR L (Attention Layer with Residual Connection Replacement): Achieves 0.243 (E2G, 94.6% relative), 0.315 (G2E, 97.2% relative), 0.263 (E2F), 0.276 (F2E). While this is substantially better than smaller ALRR sizes—ALRR XS collapses to near-zero BLEU (0.013 on E2G, 0.010 on G2E)—the L size still falls 1–2 BLEU points short of ALR L. This gap directly quantifies the contribution of the residual connection: removing it costs approximately 1–2 BLEU points at the largest network size, and far more at smaller sizes.

• ELR L (Encoder Layer Replacement): Achieves 0.194 (E2G, 75.5% relative), 0.248 (G2E, 76.5% relative), 0.225 (E2F), 0.219 (F2E). The gap between ELR L and the baseline is approximately 5–8 BLEU points, representing a roughly 25% relative performance degradation. The paper attributes this to "the simplicity of the replacement model, which discards all of the encoder structures that aid training," though the analysis does not disentangle how much of the drop comes from removing the residual connections, layer normalization, and the position-wise feed-forward sub-layer respectively.

The size-performance scaling relationship (Figure 2) reveals an important pattern: for ALR and ASLR, performance saturates at the M size and shows minimal improvement from M to L. ALR M already achieves 0.245 (E2G), 0.320 (G2E), 0.275 (E2F), and 0.284 (F2E)—within 1–2 BLEU points of ALR L. This suggests a saturation point beyond which additional parameters yield diminishing returns. For ALRR and ELR, by contrast, performance continues to improve from M to L, with no clear saturation at the tested scale. ALRR shows a dramatic jump from ALRR M (0.158 on E2G) to ALRR L (0.243), more than doubling the BLEU score. This difference in scaling behavior between ALR/ASLR and ALRR/ELR reflects the difficulty of the learning problem: when the residual connection is preserved (ALR), the network only needs moderate capacity to learn the residual mapping; when it is removed (ALRR), substantially more capacity is needed to learn the full transformation from scratch.

The catastrophic failure of small ALRR and ELR networks deserves specific attention. At the XS and S sizes, ALRR and ELR produce BLEU scores near or at zero across all datasets. ALRR XS achieves 0.013 (E2G), 0.010 (G2E), 0.003 (E2F), and 0.001 (F2E). ELR XS is similarly collapsed: 0.012, 0.010, 0.011, and 0.011. This is not a gradual performance degradation but a phase transition: below a certain capacity threshold (somewhere between S and M), the networks cannot learn the target function at all. The paper does not investigate what happens at this threshold—whether the networks converge to a degenerate solution (e.g., always outputting the zero vector or a constant) or simply fail to fit the training data—which represents a missed opportunity for understanding the capacity requirements of attention emulation.

Full Transformer Attention Replacement: Cross-Attention Is the Hard Ceiling

The second major experimental axis tests whether the ALR approach (identified as the best encoder replacement method) generalizes to the decoder's attention blocks. Table 3 reports results for five configurations: decoder self-attention only (Dec SA), decoder cross-attention only (Dec CA), encoder and decoder self-attention jointly (E-D SA), and full replacement of all attention blocks (Full). The results are visualized in Figure 3.

Decoder self-attention replacement succeeds comparably to encoder self-attention. At the largest size (L), ALR applied to decoder self-attention achieves: 0.253 (E2G, 98.4% relative), 0.323 (G2E, 99.7% relative), 0.273 (E2F), 0.291 (F2E). These scores are essentially identical to the ALR L encoder self-attention results. This demonstrates that the feed-forward networks can handle the causal masking constraint of decoder self-attention—the autoregressive constraint where each position can only attend to previous positions—as effectively as they handle the bidirectional context of encoder self-attention. The position-by-position processing procedure described in Section 3.4 (Appendix B) successfully replicates the masked self-attention behavior.

Encoder and decoder self-attention jointly (E-D SA) show modest compounding of degradation. With ALR L replacing both encoder and decoder self-attention (but leaving cross-attention intact), BLEU scores are: 0.246 (E2G, 95.7% relative), 0.321 (G2E, 99.1% relative), 0.270 (E2F), 0.284 (F2E). These are approximately 1 BLEU point lower than replacing only encoder or only decoder self-attention, suggesting a small compounding effect where errors from the encoder replacement propagate through the decoder replacement.

Decoder cross-attention replacement collapses performance to near-random levels. This is the paper's most important negative result. With ALR L replacing decoder cross-attention only (and leaving all self-attention intact), BLEU scores drop to: 0.115 (E2G, 44.7% relative), 0.130 (G2E, 40.1% relative), 0.109 (E2F), 0.115 (F2E). Performance degrades further at smaller sizes: ALR M achieves only 0.104 (E2G), and ALR S drops to 0.054 (E2G). The trend from XS to L shows steady improvement with size (XS: 0.035, S: 0.054, M: 0.104, L: 0.115 on E2G), indicating that larger networks help but do not close the gap. Even at L size, cross-attention replacement retains less than half of the baseline BLEU.

Full attention replacement is bottlenecked by cross-attention. When all attention blocks are replaced (Full L), the scores closely track the cross-attention-only scores: 0.105 (E2G), 0.134 (G2E), 0.117 (E2F), 0.116 (F2E). This confirms that cross-attention is the dominant bottleneck: even though encoder and decoder self-attention can be replaced successfully, the failure of cross-attention replacement drags the full model's performance down to cross-attention-only levels. The full replacement scores are slightly different from cross-attention-only scores (e.g., 0.105 vs. 0.115 on E2G at size L), indicating minor interactions between the replaced components, but the dominant signal is clear: cross-attention resistance determines the ceiling.

Size scaling for cross-attention replacement. Unlike self-attention replacement, where performance saturates at M or L sizes, cross-attention replacement shows no sign of saturation at the largest tested size. The trajectory from XS (0.035) to S (0.054) to M (0.104) to L (0.115) is still improving at L, though the rate of improvement slows. The paper does not test sizes larger than L, so it is unknown whether further scaling would eventually close the gap or whether cross-attention replacement asymptotes below baseline performance.

Cross-Dataset Consistency

The results are remarkably consistent across all four language pairs. The relative ordering of methods (ALR ≈ ASLR > ALRR > ELR for encoder; Dec SA ≈ Enc SA >> Dec CA for attention types) holds for every language pair without exception. The German-English pair (G2E) consistently shows the highest BLEU scores and English-German (E2G) the lowest, following the teacher Transformer's baseline pattern (Table 2). No anomalous results or dataset-specific failures are observed. This consistency strengthens the paper's claims of robustness across languages, though all four pairs share the same dataset characteristics (IWSLT2017, spoken language translation, similar training set sizes).

Ablation Studies and Robustness Checks

The paper's experimental design embeds several structural comparisons that function as ablations, though they are not labeled as such in the text. Each comparison isolates a specific design dimension:

• Residual connection contribution (ALR vs. ALRR): Holding the replacement level constant (replacing the attention block) while varying whether the residual connection is preserved (ALR) or removed (ALRR) isolates the residual pathway's contribution. At size L, ALR achieves relative BLEU of ~98–100% across language pairs while ALRR achieves ~94–97% (Table 3), quantifying the residual connection as worth approximately 1–3 relative percentage points at sufficient capacity. At smaller sizes, the gap is much larger: ALR S achieves relative scores of ~76–88% (E2G: 0.196 vs. 0.257 baseline, 76.3% relative) while ALRR S collapses to near zero (0.018 on E2G, 7.0% relative). This interaction between size and residual benefit is not discussed in the paper but is a notable empirical finding: the residual connection's value is most pronounced at low capacity, suggesting that it primarily aids learnability rather than providing representational benefits that large networks can internalize.

• Multi-head structure contribution (ALR vs. ASLR): The comparison between a single monolithic replacement network (ALR) and per-head replacement networks (ASLR) tests whether the multi-head decomposition matters. At size L, ASLR achieves relative BLEU of ~98–100% (Table 3), essentially identical to ALR L. Even at size XS, ASLR XS (0.245 E2G) performs similarly to ALR M (0.245) despite having fewer parameters per head (290K for ASLR XS per head vs. 10M for ALR M total—though the total parameter comparison depends on the number of heads, which is not specified). The paper does not draw strong conclusions from this, but the implication is that the multi-head structure is not a necessary inductive bias for representing or learning the self-attention function at this scale. The fact that ASLR does not outperform ALR, despite having a structure more closely matching the original attention, suggests that the head-wise decomposition provides no meaningful advantage for distillation-based learning.

• Layer structure contribution (ALRR vs. ELR): The comparison between replacing attention plus residual (ALRR) and replacing the entire encoder layer (ELR) isolates the value of the remaining architectural components—layer normalization and the position-wise feed-forward sub-layer. At size L, ALRR L achieves 0.243 (E2G) while ELR L achieves 0.194, a gap of approximately 0.05 BLEU or 5 percentage points. At size M, the gap is smaller in absolute terms: ALRR M achieves 0.158 while ELR M achieves 0.116, a gap of 0.04 BLEU. This comparison indicates that even when attention and residual connections are removed, the layer normalization and feed-forward sub-layer contribute non-trivially to the encoding function. The paper does not run ablation experiments where only layer normalization or only the feed-forward sub-layer is removed, so the contributions of these two components cannot be separated.

• Replacement scope (encoder-only vs. decoder-only vs. full): The paper systematically varies which attention blocks are replaced. The key ablation is leaving cross-attention intact while replacing self-attention everywhere (E-D SA) versus also replacing cross-attention (Full). At size L, E-D SA achieves 0.246 (E2G) while Full achieves 0.105, a drop of 0.14 BLEU that is almost entirely attributable to cross-attention replacement. This isolates cross-attention as the primary bottleneck and demonstrates that self-attention replacement can succeed in both encoder and decoder without cross-attention causing downstream degradation (since cross-attention is still functioning normally in E-D SA).

• Network size scaling (XS through L): The size sweep across four orders of magnitude (from ~300K to ~46M parameters, Table 1) serves as an implicit capacity ablation. The key finding is that different replacement variants have different scaling behaviors: ALR and ASLR saturate at M, ALRR and ELR continue improving to L, and cross-attention replacement shows slow improvement with no saturation within the tested range. This variable scaling behavior is informative about the difficulty of the respective learning problems but is not systematically analyzed in the paper.

• Negative result: Training from scratch without distillation. The paper states that "current optimization methods... are not able to train these 'attentionless Transformers' from scratch" but provides no experimental evidence for this claim. No end-to-end training experiments are reported. This is a significant gap: the claim that distillation is necessary (rather than merely helpful) is central to the paper's optimization-vs-representation argument but is asserted based on prior literature (Ba and Caruana, 2014; Urban et al., 2017) and author experience rather than demonstrated experimentally. An experiment attempting to train even one of the replacement variants (e.g., ALR M) end-to-end with the translation objective—and showing that it fails to converge or substantially underperforms the distilled version—would have strengthened this claim considerably.

Critical Assessment

The paper makes one central empirical claim: shallow feed-forward networks trained via knowledge distillation can replace self-attention in the Transformer with minimal performance degradation, but cannot effectively replace cross-attention. Let us examine the evidence for and against each component of this claim, and identify what the experiments do and do not demonstrate.

Can feed-forward networks replace self-attention? The evidence for encoder self-attention replacement is strong within the paper's specific experimental conditions. ALR L achieves BLEU scores within 1–2 points of the teacher Transformer on all four language pairs (Table 3). ALR M achieves scores within 1–3 points. The results are consistent across language pairs and across encoder and decoder self-attention. The size scaling curves (Figure 2) suggest that performance saturates at or near teacher performance at larger sizes, indicating that the feed-forward networks are not merely approximating the function but can match it closely enough to maintain translation quality.

However, what the experiments do not demonstrate is also important. They do not show that the replacement is practical: the replacement networks are 5× to 700× larger than the attention layer they replace (320K–41M vs. 60K parameters per layer), so no efficiency gain is achieved. They do not show that the replacement generalizes: only one dataset (IWSLT2017) and one model scale (128-dimensional embedding) are tested. The 128-dimensional Transformer is substantially smaller than production translation models, and it is unknown whether the results would hold at larger scales (512-dim, 1024-dim) where the attention function to be emulated may be more complex. They do not show that the replacement is necessary: there is no demonstration that the function cannot be learned some other way. And critically, they do not show that the replacement preserves all properties of attention: the fixed-size input architecture eliminates the Transformer's ability to handle variable-length sequences, which is one of attention's primary practical advantages.

Can feed-forward networks replace cross-attention? The negative result is the paper's most robust and informative finding. Cross-attention replacement fails dramatically, with ALR L achieving only 40–45% of baseline BLEU (Table 3: Dec CA L scores of 0.115–0.130 vs. baselines of 0.257–0.324). This failure is consistent across all four language pairs, all network sizes tested, and whether cross-attention is replaced alone or alongside self-attention. The failure is not subtle or borderline—it is a clear performance cliff.

What the experiments do not definitively establish is why cross-attention replacement fails. The paper's hypothesis—"the proposed shallow networks were not able to capture the more intricate and complex interactions between the differing sequences"—is plausible but untested. Alternative explanations that the experiments cannot rule out include: (a) the cross-attention network has twice the input dimension (12,800 vs. 6,400) and may simply need even larger hidden layers—the size sweep does not go far enough to reach saturation; (b) the concatenation of encoder and decoder representations into a flat vector may create an input representation that is inherently harder to learn from, independent of the underlying function's complexity; (c) the distillation data for cross-attention may be of lower effective quality—the cross-attention activations may vary more across sentences, requiring more training examples than the 200,000-sentence IWSLT2017 training set provides; (d) the fixed-size padding approach may interact particularly badly with cross-attention because the alignment between source and target positions varies sentence-by-sentence, and the fixed mapping cannot capture this alignment variability. Each of these explanations suggests a different remedy (more capacity, different input representation, more data, sequence-length-agnostic architectures) and the paper provides no evidence distinguishing among them.

Does the paper demonstrate that attention is an "optimization scaffold" rather than a "representational necessity"? This is the paper's most conceptually ambitious claim and the one with the weakest experimental support. The argument rests on two premises: (1) feed-forward networks can represent the self-attention function (demonstrated by successful distillation), and (2) feed-forward networks cannot learn this function through end-to-end training (asserted but not demonstrated). Only premise (1) is evidenced in the paper. Premise (2) is supported by citation to the broader distillation literature (Ba and Caruana, 2014; Urban et al., 2017) and by the paper's concluding statement, but no experiment in this paper tests end-to-end training of an attentionless Transformer. This is a significant gap because premise (2) is logically necessary for the optimization-scaffold interpretation. Without it, an alternative interpretation is equally consistent with the data: that feed-forward networks can represent and can learn the function end-to-end, and distillation is simply a more efficient training method but not a strict necessity. The paper's contribution would be stronger if it included even a single experiment attempting end-to-end training of one replacement variant (e.g., ALR M) and showing failure or substantial underperformance.

Does the paper demonstrate that the residual connection's benefit is primarily optimizational? The comparison between ALR (residual preserved) and ALRR (residual removed) provides partial evidence. The fact that ALRR can achieve near-baseline performance at size L (0.243 on E2G vs. 0.257 baseline) shows that the residual connection is not strictly necessary for representing the self-attention function—a sufficiently large feed-forward network can internalize the residual computation. However, the dramatic failure of ALRR at small sizes (ALRR XS: 0.013 E2G) while ALR XS achieves reasonable performance (0.180 E2G) suggests that the residual connection provides a strong inductive bias that makes the function learnable with less capacity. This is consistent with an optimization-scaffold interpretation: the residual connection creates a simpler loss landscape where smaller networks can succeed. But again, this evidence comes entirely from the distillation setting, not from end-to-end training, so the optimization story is about the distillation objective's optimization landscape, not necessarily the translation objective's optimization landscape.

Missing experiments that would strengthen the paper. Several additional experiments would substantially increase confidence in the paper's findings:

• End-to-end training of at least one replacement variant (e.g., ALR M) with the translation objective, to test whether distillation is truly necessary or merely helpful. A negative result (failure to converge or poor BLEU) would provide direct evidence for the optimization-scaffold hypothesis. A positive result would substantially change the paper's narrative.

• Scale to larger model dimensions. The 128-dimensional embedding is a deliberate reduction from the standard 512. Testing ALR at 512 dimensions would address whether the replacement success is specific to small models or generalizes to more realistic scales.

• Tests on additional datasets and language pairs. IWSLT2017 is a single domain (TED talks) with relatively short sentences. Testing on WMT news translation (longer sentences, more formal domain) would test whether the fixed-size padding approach scales to longer sequences and whether the findings are domain-specific.

• Inference speed and memory benchmarks. The paper acknowledges the parameter-count cost but provides no runtime measurements. Knowing the wall-clock inference time and memory usage of the attentionless Transformer versus the original would clarify whether there are any practical scenarios where this approach might be preferred despite the parameter inflation.

• A cross-attention capacity saturation experiment. Testing cross-attention replacement at sizes larger than L would reveal whether the performance gap eventually closes with sufficient capacity, or whether cross-attention replacement asymptotes below baseline regardless of scale. This would distinguish between "cross-attention is harder to learn" and "cross-attention cannot be represented by this architecture."

• Varying the maximum sequence length. The fixed length of 50 is chosen to cover 96% of data. Testing whether performance degrades as the maximum length increases (forcing the replacement network to handle a larger fixed input) or decreases would characterize the sensitivity of the approach to sequence length.

What the experiments do and do not establish, in summary. The experiments convincingly establish that shallow feed-forward networks can, through knowledge distillation from a trained Transformer, learn to replicate the input-output behavior of self-attention layers with sufficient fidelity to maintain translation quality on IWSLT2017 using a 128-dimensional model. They equally convincingly establish that cross-attention replacement under the same conditions is substantially harder and does not succeed at the tested scales. These are novel, specific, and well-supported empirical findings.

The experiments do not establish that attention is generally unnecessary for sequence-to-sequence tasks, that the replacement approach is practical, that distillation is the only viable training method, that the findings scale to larger models or other tasks, or that the optimization-scaffold interpretation (as opposed to the simpler "distillation is a powerful training technique" interpretation) is correct. The paper's conceptual claims outrun its experimental evidence in several places, particularly in the conclusion's suggestion that future optimization advances could make "less specialized architectures such as feed-forward networks" viable for tasks "currently reserved for highly specialized architectures." This is an interesting speculation motivated by the results but not directly supported by them—the experiments show what is possible with distillation, not what might be possible with better optimization.

6. Limitations and Trade-offs

6.1 The Fixed Sequence Length Constraint Eliminates the Transformer's Primary Practical Advantage

The assumption or constraint. The replacement approach requires all input sentences to be padded or truncated to a fixed maximum length (50 tokens) because the feed-forward replacement networks accept only fixed-size input vectors. The paper explicitly acknowledges this trade-off:

"Another downside of our replacement of the attention with a fixed-size feed-forward network is the imminent lack of flexibility of the model in terms of the length of sequences the model can operate with."

This is not an implementation detail but a fundamental architectural choice: the concatenation-based input representation (all 50 word embeddings flattened into one 6,400-dimensional vector for self-attention, 12,800 for cross-attention) means the network's input dimensionality, its first-layer weight matrix, and its entire learned mapping are hard-coded to exactly 50 positions. There is no mechanism for handling shorter sequences without padding or longer sequences without truncation.

The consequence. The resulting "attentionless Transformer" loses what is arguably the Transformer's most important practical advantage over previous architectures: the ability to process variable-length sequences with a fixed set of parameters. The original Transformer can handle a sentence of any length (up to memory constraints) because attention applies the same position-invariant computation at every token position. The replacement model cannot process a 51-word sentence at all—it would require a completely different network architecture with a larger input layer. This means the approach cannot be deployed in any setting where sequence lengths vary unpredictably beyond the training-time maximum, which includes essentially all real-world translation applications.

Additionally, the 50-token limit is not a fundamental constant but a dataset-specific choice justified by coverage: "96% of the samples in the datasets are of length 50 or less" (Appendix B). For IWSLT2017 (TED talks with relatively short utterances), this truncation discards only 4% of training data. For domains with longer typical sequences—news translation, document-level translation, legal or medical text—the coverage at length 50 would be much lower, and the truncation loss would be proportionally larger. The approach does not scale gracefully to longer sequences because the input dimensionality grows linearly with the maximum length, causing the first-layer parameter count to grow quadratically (input_dim × hidden_dim). A 100-token maximum would double the input size to 12,800 and roughly double the parameter count.

What evidence exists in the paper. The fixed-length constraint is not experimentally ablated. The paper does not test the sensitivity of performance to the maximum length choice (e.g., comparing max length 30 vs. 50 vs. 70), does not report how performance degrades for sentences near the maximum length versus short sentences, and does not attempt architectures that might handle variable-length sequences (e.g., recurrent or 1D-convolutional networks that process tokens independently). The effect of the 4% data loss from truncation on baseline performance is not quantified.

Mitigation status. The paper acknowledges this limitation explicitly in the Discussion section but does not propose or test any mitigation. There is no discussion of potential solutions (e.g., using a recurrent or convolutional architecture that shares parameters across positions, or training multiple networks for different length buckets). The limitation is presented as an inherent trade-off of the fixed-size feed-forward approach rather than as a problem to be solved.

---

6.2 The Parameter Cost Is Dramatically Higher Than Attention, Negating Any Practical Efficiency Argument

The assumption or constraint. The replacement networks achieve comparable performance to attention only through a massive increase in parameter count. The paper documents this explicitly in Table 1: the original attention layer has approximately 60,000 parameters, while the replacement feed-forward networks range from 320,000 (ALR XS, 5.3× larger) to 41,000,000 (ALR L, 683× larger). The paper attributes this to the fixed-size input format:

"The original number of parameters for the attention layer (60,000 parameters in our case) is mostly exceeded by the replacement networks, mainly due to fixed-size inputs and outputs, and the processing format demanded by the FF network."

Even at the smallest size (XS), which achieves substantially degraded performance (ALR Enc SA XS: BLEU 0.180 on E2G vs. 0.257 baseline, a 30% relative drop), the replacement is 5× larger than attention. At the M size, which approaches but does not fully match baseline (0.245 vs. 0.257), the replacement is ~167× larger. The largest size (L, 41M parameters), which matches baseline performance, is nearly 700× larger.

The consequence. This parameter inflation means the replacement approach offers no practical advantage over attention for any deployment scenario. The headline result—"attentionless Transformers rival the performance of the original architecture"—is true only when the replacement model is allowed to use orders of magnitude more parameters. For a practitioner deciding whether to use this method, the trade-off is unambiguous: for the same or higher parameter cost, you can have a standard Transformer that handles variable-length sequences natively and benefits from decades of optimization work (efficient attention kernels, FlashAttention, etc.), or you can have an "attentionless" version that is strictly less flexible, likely slower (the inference procedure for decoder self-attention runs the FF network once per position, as described in Appendix B), and no more accurate.

The parameter comparison is also inexact in a way that understates the gap. The 60,000-parameter figure counts only the multi-head attention projection matrices (Q, K, V, and output projections) for a single layer. The replacement networks, by contrast, include both the attention computation's parameters and the implicit positional processing that in a standard Transformer is handled by the (parameter-free) dot-product attention mechanism plus positional encodings. The replacement networks must learn position-specific transformations because the concatenated input has a fixed spatial structure, while attention achieves position-equivariance with zero additional parameters. So the 683× multiplier for ALR L represents not just replacing the attention computation but also buying back the inductive biases that attention provides for free.

What evidence exists in the paper. Table 1 provides the parameter counts. The paper does not run any compute-matched or parameter-matched comparisons—for example, scaling the replacement network down to 60,000 parameters to see what BLEU the approach achieves at equal cost, or scaling the attention layer up to match the replacement's parameter count to establish an upper bound on what additional attention parameters would buy. The paper does not measure inference latency, memory usage, or training time, so the practical cost in wall-clock and hardware terms is unknown beyond the raw parameter counts.

Mitigation status. The paper acknowledges the parameter cost in the Discussion section ("all of the replacement approaches come at a significant cost of having more parameters") and in Appendix D, where the authors suggest that "further optimization of the FF networks' hyperparameters using advanced parameter search... could yield even better results in terms of translation quality and possibly even enable the usage of smaller FF networks for the replacement, as the size of the networks represents one of the major bottlenecks for the deployment of these 'attentionless' Transformers in practice." However, no experiments toward this goal are reported, and the suggestion that hyperparameter tuning could close a 683× parameter gap is optimistic at best. The fundamental driver of parameter count—the fixed-size concatenated input creating an enormous first-layer weight matrix—is architectural, not a matter of learning rate or batch size.

---

6.3 Cross-Attention Replacement Fails, Which Caps the Scope to Encoder-Only or Self-Attention-Only Substitution

The assumption or constraint. The paper's approach fundamentally fails to replace the decoder's cross-attention mechanism, limiting the "attentionless Transformer" concept to architectures that either (a) only need self-attention (encoder-only models, decoder-only language models) or (b) retain cross-attention in their original form. The cross-attention replacement collapses BLEU scores to roughly 35–45% of baseline even at the largest tested network size (Table 3: Dec CA L achieves 0.115 on E2G vs. 0.257 baseline; 0.130 on G2E vs. 0.324 baseline). The paper hypothesizes:

"This suggests that the proposed shallow networks were not able to capture the more intricate and complex interactions between the differing sequences that enter the cross-attention mechanism."

The consequence. The "fully attentionless Transformer" that the paper's title and abstract gesture toward is not achievable with the proposed methods. The full replacement results (Table 3, "Full" rows) are bottlenecked by cross-attention: Full L achieves 0.105 (E2G), 0.134 (G2E), 0.117 (E2F), 0.116 (F2E)—essentially identical to replacing cross-attention alone. This means the approach can only replace self-attention in the Transformer, not attention in general. The decoder's cross-attention, which is responsible for the critical task of aligning source and target language representations during generation, remains indispensable within this framework.

This restriction significantly narrows the paper's applicability claims. The Transformer's three attention mechanisms play different roles, and the paper shows that two of the three (encoder self-attention, decoder self-attention) are replaceable with sufficient parameters while one (cross-attention) is not at the tested scales. This undermines any general claim that "Transformers do not necessarily need to have attention" (Conclusion)—they still need cross-attention for sequence-to-sequence tasks, which is the setting the entire paper operates in. The statement is only accurate for the self-attention components.

For practitioners, this means the "attentionless" approach is most applicable to encoder-only tasks (classification, representation learning) or decoder-only language modeling, where cross-attention does not exist. For machine translation specifically, the practical path suggested by the paper's results would be a hybrid model: replace self-attention with feed-forward networks but retain standard cross-attention. Such a model would still incur the fixed-length constraint and parameter inflation penalties described in Limitations 6.1 and 6.2, while retaining the quadratic complexity of cross-attention—a worst-of-both-worlds outcome.

What evidence exists in the paper. The cross-attention failure is the most thoroughly evidenced negative result in the paper, appearing across all four language pairs, all network sizes (XS through L), and both cross-attention-only and full-replacement configurations (Table 3). The size scaling for cross-attention replacement (XS: 0.035, S: 0.054, M: 0.104, L: 0.115 on E2G) shows steady improvement but no sign of saturation at L, leaving open the possibility that even larger networks might eventually close the gap. However, the paper tests no sizes beyond L (41M parameters per cross-attention network, with 12,800-dimensional input), so whether cross-attention replacement can work at all is unknown.

Mitigation status. The paper proposes no specific mitigation for this failure, beyond the general suggestion in Appendix D that "another potential direction lies in the training of more complex FF networks for the purpose of modeling the cross-attention module of the decoder, as the current shallow network shows that, in contrast to self-attention which they can learn successfully, cross-attention proves to be more challenging due to its complexity." What "more complex FF networks" would entail—deeper networks, different architectures, different input representations—is not specified or tested. The paper provides no diagnostic analysis of how cross-attention replacement fails (e.g., whether the networks fail to learn the alignment function at all, or learn it but with insufficient precision), which would be necessary to guide mitigation efforts.

---

6.4 The Necessity of Knowledge Distillation Is Asserted but Not Experimentally Demonstrated

The assumption or constraint. The paper's central conceptual claim—that attention serves as an optimization scaffold rather than a representational necessity—rests on the premise that feed-forward replacements can represent the attention function (shown via distillation) but cannot learn it through end-to-end training (not shown). The paper states this conclusion explicitly:

"These conclusions also point out the deficiencies of the current optimization methods, which are not able to train these 'attentionless Transformers' from scratch but need more advanced techniques, such as knowledge distillation to converge into desired parameter configurations."

However, the paper reports no end-to-end training experiments whatsoever. No variant of the attentionless Transformer is trained from scratch with the translation objective to test whether distillation is truly necessary or merely helpful.

The consequence. Without an end-to-end training baseline, the paper's optimization-scaffold interpretation is a hypothesis, not a demonstrated finding. Several alternative explanations for why distillation was used are equally consistent with the data and cannot be ruled out:

1. Distillation may be sufficient but not necessary—end-to-end training might work but take longer or require more hyperparameter tuning. The authors' choice of distillation could reflect convenience rather than necessity.

2. Distillation may provide a better initialization—the replacement networks could potentially be fine-tuned end-to-end after distillation to further improve performance, but the paper does not test this.

3. The failure of end-to-end training, if it were demonstrated, might reflect hyperparameter mismatch rather than a fundamental optimization barrier. The training settings used for the standard Transformer (which are not reported in detail) may simply be inappropriate for the attentionless architecture, and a proper hyperparameter search might succeed.

4. The attentionless architecture may have different convergence rates—it might eventually reach comparable performance with longer training, more data, or different optimization algorithms. The paper's claim that current methods "are not able to train" these models is an absolute statement that would require showing failure under extensive optimization attempts.

The practical consequence is that a practitioner cannot know whether an attentionless Transformer could be trained end-to-end for a new task without a pre-trained teacher model available. If distillation is truly necessary, the approach requires an already-trained Transformer for every new language pair or domain—you cannot train an attentionless model from scratch on novel data. If distillation is merely helpful, the approach has broader applicability but the paper provides no guidance on how to train these models without a teacher.

What evidence exists in the paper. None. The paper cites Ba and Caruana (2014) and Urban et al. (2017) for the general concept of using distillation to train simpler models, but those papers address different architectures (CNNs) on different tasks (image classification). Whether their findings about the necessity of distillation transfer to attention replacement in Transformers is unknown. The paper provides no ablation where even one replacement variant is trained end-to-end, which would be the most direct test of the optimization-scaffold hypothesis.

Mitigation status. Not addressed. The paper treats distillation as a methodological choice rather than a hypothesis to be tested, and the strong claims in the Conclusion about optimization deficiencies are presented as implications rather than as experimentally supported conclusions. The authors do not flag the absence of end-to-end experiments as a limitation or suggest it as future work.

---

6.5 Generalisation Is Untested: Single Dataset, Single Model Scale, Single Task Family

The assumption or constraint. All experiments are conducted on a single dataset (IWSLT2017, a spoken language translation benchmark derived from TED talks) using a single model configuration (a 128-dimensional Transformer, deliberately reduced from the standard 512 dimensions for computational reasons) and a single task family (machine translation). The paper provides no evidence about whether the feed-forward replacement approach transfers to:

• Other sequence-to-sequence tasks such as text summarization, grammatical error correction, or dialogue generation, which have different output characteristics and may place different demands on attention mechanisms.
• Other model scales, particularly the 512-dimensional or larger Transformers used in production systems. The paper acknowledges that the embedding size reduction "did not drop the overall score too much below the original BLEU score but resulted in significantly lower computational power demands," but provides no comparison to quantify the performance impact or to test whether the replacement approach works at standard scales.
• Other domains with longer sequences, more formal language, or different linguistic properties. IWSLT2017's TED talks represent a specific genre with relatively short, spoken-language utterances.
• Encoder-only or decoder-only architectures such as BERT or GPT-style models. The paper's framing suggests these would be natural applications (since they only contain self-attention, which the approach successfully replaces), but no experiments are conducted.
• Other language pairs beyond the four tested, particularly non-European languages with different word order, morphology, or script systems.

The consequence. The paper's claims are, strictly speaking, valid only for the specific configuration tested: a 128-dimensional Transformer trained on IWSLT2017 translation. Whether the findings reflect general properties of attention mechanisms or specific properties of this dataset and model scale is unknown. Several aspects of the experimental setup could drive the results in ways that would not transfer:

• The 96% coverage at max length 50 is specific to IWSLT2017's sentence length distribution. For a dataset with longer average sentences, the fixed-size constraint would require either a larger maximum length (increasing parameter count further) or more aggressive truncation (losing more training data).
• The 128-dimensional embedding means the attention mechanism has relatively few parameters to learn a relatively simple function—the function may be easier to emulate at this small scale than at 512 or 1024 dimensions, where attention has more capacity to learn complex, input-dependent patterns that a feed-forward network might struggle to capture.
• The baseline BLEU scores (0.257–0.324) are modest by modern standards. A stronger baseline Transformer might leave less room for the replacement to match performance, or might exhibit attention behaviors that are harder to emulate.

What evidence exists in the paper. The paper reports results on four language pairs within IWSLT2017, which provides within-dataset consistency evidence (the relative ordering of methods is stable across language pairs) but does not constitute a cross-domain or cross-task generalization test. No experiments on other datasets, tasks, or model scales are reported or even discussed as future work in the main text.

Mitigation status. The paper does not acknowledge this as a limitation. The abstract claims the experiments "reveal the capacity of these 'attentionless Transformers' to rival the performance of the original architecture" without qualification about the specific dataset or model scale. The generality of this claim is not supported by the experimental design. The limitation is implicit in the experimental setup but is never explicitly discussed.

---

6.6 The Difficulty Estimation Cost for Cross-Attention's Failure Is Unanalyzed, and No Diagnostic Experiments Identify the Root Cause

The assumption or constraint. The paper identifies cross-attention replacement as the primary failure mode but performs no diagnostic analysis to characterize how or why it fails. The only analysis offered is the brief hypothesis quoted in Limitation 6.3 about "more intricate and complex interactions." The experiments report BLEU scores at different network sizes (Table 3) and show that cross-attention replacement underperforms, but provide no evidence about:

• Whether the cross-attention replacement networks fail to learn the teacher's activation patterns at all (high training loss) or learn them but produce activations that don't support translation (low training loss but poor downstream performance).
• Whether the failure is uniform across sentence lengths or concentrated in longer sentences where the alignment problem is more complex.
• Whether the concatenated input representation (encoder + decoder states flattened together) creates an inherently ambiguous mapping that the network cannot disambiguate.
• Whether larger networks continue to improve cross-attention BLEU or approach an asymptote below baseline—the size scaling from XS to L shows monotonic improvement, but the rate of improvement is slowing and the L size is far from baseline.
• Whether the failure is specific to the one-hidden-layer architecture or would persist with deeper networks, different activation functions, or different input representations.

The consequence. Without understanding why cross-attention replacement fails, the paper cannot provide guidance on whether the failure is fundamental (cross-attention performs a computation that feed-forward networks of any reasonable size cannot represent) or contingent (the specific architecture, training procedure, or data representation is inadequate but could be fixed). This matters for the paper's broader narrative: if the failure is fundamental, it establishes an important boundary on what attention contributes that simpler architectures cannot replicate. If the failure is contingent, it is merely an implementation limitation that future engineering could overcome. The paper's framing leans toward the fundamental interpretation ("more intricate and complex interactions") but provides no discrimination between the two possibilities.

For a practitioner, this diagnostic gap means that attempting to improve cross-attention replacement would require starting from scratch—testing hypotheses about input representation, network depth, training data requirements, and capacity scaling without any experimental guidance from the paper about which direction is promising.

What evidence exists in the paper. The only evidence about cross-attention replacement is the BLEU scores in Table 3 and Figure 3. The paper reports no training loss curves for the cross-attention replacement networks (which would show whether they successfully fit the teacher's activations), no analysis of which types of sentences or translation phenomena suffer most from cross-attention replacement, and no ablation of alternative cross-attention replacement architectures (e.g., deeper networks, separate processing of encoder and decoder states before combination, attention-like structures that handle alignment explicitly). The cross-attention networks have twice the input dimension of self-attention networks (12,800 vs. 6,400) due to concatenating encoder and decoder states, but whether this factor alone explains the failure is not investigated.

Mitigation status. The paper acknowledges in Appendix D that cross-attention replacement is the key unsolved problem: "another potential direction lies in the training of more complex FF networks for the purpose of modeling the cross-attention module of the decoder." However, the experiments provide no starting point for this future work. The failure mode remains a black box: cross-attention replacement produces poor BLEU scores, and we don't know why.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new state-of-the-art model or a practical deployment strategy—the replacement networks are orders of magnitude larger than the attention layers they replace, and the fixed-length constraint makes them strictly less flexible than the original Transformer. Its contribution is instead diagnostic and conceptual: it provides the first systematic evidence that multi-head self-attention can be functionally replaced by a generic feed-forward network through knowledge distillation, while cross-attention cannot (at the tested scales). This finding shifts how the field should think about what attention contributes to the Transformer architecture.

Attention's role is refined from "representational necessity" to "optimization scaffold." Prior to this work, the dominant understanding—implicit in the architecture's design and in most follow-up work—was that attention's dynamic, content-dependent routing (pairwise dot-product similarity, softmax normalization, weighted aggregation) provides a representational capability that simpler architectures lack: the ability to selectively attend to relevant parts of a sequence based on the input itself. This paper demonstrates that, at least for self-attention in a 128-dimensional Transformer on IWSLT2017 translation, a static feed-forward mapping with no dynamic routing can achieve the same input-output behavior when trained via distillation (ALR L achieves 0.252 BLEU vs. 0.257 baseline on English-German; Table 3). This means the representational argument for self-attention—that its specific computation is architecturally required to capture the needed sequence transformations—is empirically falsified for this setting.

What the paper puts in place of the representational-necessity view is an optimization-scaffold hypothesis: self-attention's specific computational structure (query-key-value projections, scaled dot-product attention, softmax) creates a loss landscape where gradient descent reliably finds good parameters, while a generic feed-forward network—though formally capable of representing the same function—cannot discover that function through end-to-end training with current optimization methods. The paper's concluding statement captures this:

"These conclusions also point out the deficiencies of the current optimization methods, which are not able to train these 'attentionless Transformers' from scratch but need more advanced techniques, such as knowledge distillation to converge into desired parameter configurations."

This is a conceptual reframing of moderate significance, not a paradigm shift. It does not overturn the Transformer architecture or render attention obsolete—the replacement networks are impractical, and attention remains the only known way to achieve this performance with reasonable parameter counts and without distillation. But it changes the conversation about why attention works from one about representational capacity to one about learnability. This connects the Transformer literature to broader themes in deep learning about the relationship between architecture and optimization: the lottery ticket hypothesis (Frankle and Carbin, 2019), the role of inductive biases in making functions discoverable rather than merely representable, and the growing recognition that architecture design is often optimization engineering.

The work reconciles a latent tension in the knowledge distillation literature. Prior distillation work (Ba and Caruana, 2014; Urban et al., 2017) showed that deep convolutional networks could be compressed into shallower architectures and that even the convolutional inductive bias could be discarded. But those results applied to vision tasks where the target architecture (CNNs) and the replacement (MLPs) both operated on fixed-size inputs. This paper extends the question to sequence models, where the input is inherently variable-length and the architecture must handle position-invariant processing—and it shows that the answer is partially yes (self-attention) and partially no (cross-attention). This resolves the open question of whether the distillation-to-simpler-architecture paradigm transfers to attention-based models, and it does so with a nuanced answer that depends on which attention mechanism is being replaced. The tension between "Transformers need attention" (the architectural orthodoxy) and "deep networks can be compressed into shallow ones" (the distillation literature) is resolved into a conditional statement: self-attention is compressible through distillation; cross-attention, at tested scales, is not.

The paper establishes a new diagnostic methodology for architectural analysis. The component-level replacement paradigm—replacing individual sub-layers with architecturally unrelated networks trained via distillation, varying the level of abstraction to localize functional contributions—provides a template for future architectural investigations that is more informative than standard ablation (which only tests necessity by removing a component) and more targeted than whole-model compression (which cannot isolate the role of specific mechanisms). The four-variant hierarchy (ALR → ALRR → ASLR → ELR) demonstrates how graduated replacement levels can incrementally characterize which structural elements matter and in what way. This methodology could be applied to investigate other architectural questions: the role of gating in LSTMs, the necessity of depthwise separable convolutions in MobileNets, the contribution of specific normalization schemes. The key elements are (1) replacement by a generic architecture to eliminate inductive bias, (2) distillation to factor out optimization effects, and (3) systematically varying the replacement boundary to localize contributions.

Research directions that become more attractive. The paper's positive result for self-attention replacement makes encoder-only and decoder-only language model simplification an attractive target—these architectures don't contain cross-attention, so the primary failure mode is absent. The negative result for cross-attention makes understanding cross-attention's specific contribution a more urgent research question than it was before: if self-attention is emulatable but cross-attention is not, what specific computation does cross-attention perform that resists simplification? The parameter inflation documented in Table 1 (5× to 700× overhead) makes architectural efficiency quantification a newly tractable research direction—the overhead ratio provides a concrete metric for how much an architectural prior is "worth" in representational terms.

Research directions that become less attractive. The paper's finding that the replacement networks are substantially larger than attention (not smaller) and that they cannot handle variable-length sequences suggests that naïve attention replacement for model compression is not a promising direction without fundamental changes to the replacement architecture (e.g., parameter sharing across positions). The failure of end-to-end training for these architectures (asserted by the paper, though not experimentally demonstrated) also suggests that direct optimization of attentionless Transformers from scratch is unlikely to succeed without advances in optimization methods, making it a high-risk research direction.

---

Follow-Up Research This Work Enables

End-to-end training of attentionless Transformers: testing the optimization-scaffold hypothesis directly. The paper's central conceptual claim—that attention provides an optimization scaffold rather than a representational necessity—rests on an assertion that end-to-end training of attentionless Transformers fails. But the paper reports no such experiment, citing prior work and author experience. A direct follow-up would train one of the successful replacement architectures (e.g., ALR M, which achieves near-baseline BLEU when distilled) end-to-end from scratch on the IWSLT2017 translation task, with extensive hyperparameter tuning (learning rate schedules, warmup, gradient clipping, optimizer variants—AdamW, LAMB, Shampoo) and substantially longer training than the standard Transformer receives. The specific experiment: initialize an ALR M attentionless Transformer randomly, train it on the same IWSLT2017 data with the same translation loss as the teacher, and measure whether it (a) fails entirely (BLEU near zero, confirming the optimization-scaffold hypothesis), (b) converges but to lower performance than the distilled version (consistent with a partial optimization-scaffold effect), or (c) matches the distilled version given enough tuning and training (falsifying the hypothesis). A positive end-to-end result would substantially change the paper's narrative, and a clean negative result would strengthen it considerably. This is the single most important missing experiment in the paper.

Scaling cross-attention replacement to larger networks to determine whether the failure is a capacity ceiling or a fundamental limit. The cross-attention replacement BLEU scores show monotonic improvement from XS (0.035 on E2G) through S (0.054) to M (0.104) to L (0.115), with no clear saturation at L. This leaves open the question: if the cross-attention replacement networks were scaled further (e.g., 2× L, 4× L, up to GPT-2-scale parameter counts), would the BLEU gap eventually close, or does it asymptote at some value substantially below baseline? This question distinguishes between two fundamentally different interpretations of the cross-attention failure. If the gap closes with sufficient scale, cross-attention is merely harder to emulate (requiring more capacity due to the dual-sequence input), and the finding is one of degree rather than kind. If the gap asymptotes below baseline regardless of scale, cross-attention performs a genuinely harder computation that a shallow feed-forward network of any size cannot represent—likely related to the dynamic, sentence-pair-specific alignment that cross-attention computes. The experiment: train cross-attention ALR replacement networks at sizes of 100M, 250M, and 500M parameters per layer (10× to 50× larger than the current L size) and measure whether BLEU continues to improve linearly with log-parameters or plateaus. A saturation curve would be strong evidence for a fundamental representational boundary.

Position-invariant replacement architectures to address the fixed-length constraint. The paper's replacement networks process the entire padded sentence as a single flat vector, which means (a) the model cannot handle sequences longer than the training-time maximum and (b) the parameter count scales with sequence length. A natural follow-up is to test replacement architectures that share parameters across positions, eliminating both limitations. The obvious candidate is a 1D convolutional network applied along the sequence dimension, or a recurrent network (LSTM, GRU, or a simple RNN) that processes tokens one at a time with tied weights. These architectures maintain the "shallow and generic" spirit of the investigation (no built-in attention, no dot-product similarity) while removing the fixed-length constraint. The experiment: replace each encoder self-attention block with a small 1D CNN (e.g., kernel size 3, 128 → 128 channels, 1–3 layers) or a single-layer bidirectional GRU, train via the same distillation procedure, and measure whether they match ALR's BLEU at lower parameter counts and with variable-length capability. This experiment would test whether the paper's positive self-attention result is specific to the concatenation-based input representation or generalizes to other non-attention architectures. A positive result would make the "attentionless Transformer" concept dramatically more practical; a negative result would indicate that the concatenation representation—with its position-specific processing—may itself be doing important work.

Cross-dataset and cross-scale replication to establish generality boundaries. The paper's results are from a single dataset (IWSLT2017, TED talks) with a single, deliberately reduced model scale (128-dimensional embedding). Two replication experiments would map the boundaries of the findings. First, replicate on WMT news translation (e.g., WMT14 En-De, with its longer sentences and more formal domain) to test whether the fixed-length constraint becomes a more serious limitation (WMT sentences routinely exceed 50 words) and whether self-attention replacement still succeeds in a domain where translation quality depends more heavily on long-range reordering and complex syntactic transformations. Second, replicate at standard Transformer scale (512-dimensional embedding, 8 heads, the original Vaswani et al. configuration) to test whether self-attention replacement works when the attention function to be emulated is more complex and higher-dimensional. At 512 dimensions with a 50-word maximum, the replacement network input would be 25,600 dimensions—5× larger than the current 6,400—and the parameter count for ALR M (~10M for 128-dim) would scale proportionally, likely to hundreds of millions of parameters. This experiment would reveal whether the approach scales computationally or whether the parameter inflation documented in Table 1 grows worse with model dimension, potentially making the approach infeasible at realistic scales.

Diagnostic analysis of the cross-attention failure to identify the root cause. The paper documents that cross-attention replacement fails (BLEU drops to ~40% of baseline at size L; Table 3) but provides no characterization of how it fails. A systematic diagnostic study would include: (a) Training loss analysis for the cross-attention replacement networks—do they achieve low MSE on the teacher's activations (indicating successful distillation of the mapping) or plateau at high error (indicating representational insufficiency)? If the distillation succeeds but downstream BLEU fails, the problem is in how the learned activations interact with the rest of the model. (b) Per-sentence-length analysis of cross-attention replacement BLEU—does performance degrade uniformly, or is the failure concentrated in longer sentences where the alignment problem is more complex? (c) Alignment quality analysis using word alignment metrics (e.g., Alignment Error Rate against a statistical aligner)—does the cross-attention replacement model produce poor translations because it fails to align source and target words correctly, or is the failure more diffuse? (d) Ablation of the concatenation input representation for cross-attention—test whether processing encoder and decoder states through separate feed-forward networks and then combining them (rather than concatenating into one giant vector) improves performance. Each of these diagnostics would narrow the space of hypotheses about why cross-attention fails and would guide mitigation efforts toward the right intervention (more capacity, better input representation, architectural changes, or acceptance that cross-attention is fundamentally required).

Exploring the parameter-efficiency frontier: how small can a replacement network be and still work? The paper tests five discrete sizes (XS through L) but does not systematically characterize the tradeoff between parameter count and BLEU. A parameter-efficiency study would train ALR networks at many more intermediate sizes (e.g., 100K, 200K, 400K, 800K, 1.6M, 3.2M, 6.4M, 12.8M, 25.6M) and plot the full BLEU-vs-parameters curve for encoder self-attention replacement. This curve would reveal: (a) the critical capacity threshold below which performance collapses—does the transition from non-functional to functional happen abruptly (suggesting a phase transition in representational capacity) or gradually? (b) the diminishing returns point beyond which additional parameters yield negligible BLEU improvement—the paper suggests saturation around ALR M (10M parameters), but the exact saturation point and the shape of the approach to it are not characterized. (c) Whether parameter-efficient replacements exist—is there a network size that achieves, say, 90% of baseline BLEU at less than 10× the attention layer's parameter count? This curve would provide practical guidance for anyone attempting to deploy attentionless architectures and would quantify the architectural efficiency premium of attention in a continuous rather than discrete way.

---

Practical Applications and Downstream Use Cases

The paper's approach is not currently practical for deployment—the 5× to 700× parameter inflation, the fixed-length constraint, and the cross-attention failure prevent direct application. However, the findings have indirect practical implications for specific scenarios where the diagnostic insights matter.

Guiding architecture search for on-device encoder-only models. For tasks using encoder-only Transformers (text classification, sentence embeddings, retrieval), the paper's finding that self-attention is functionally replaceable—though at high parameter cost—suggests that lightweight attention alternatives trained via distillation could be viable for deployment. The specific scenario: an organization deploys a BERT-style encoder for on-device inference where memory is limited. Standard multi-head attention has quadratic complexity in sequence length, which is unnecessary for short-text tasks (query classification, intent detection) where the attention pattern is essentially learned position-specific processing. The paper's ALR approach, modified to use a position-shared architecture (1D CNN or small RNN as proposed in the follow-up directions above) to eliminate the fixed-length constraint, could provide a drop-in replacement for the attention blocks with comparable accuracy but potentially faster inference (feed-forward and convolutional operations are more hardware-friendly than attention's gather-scatter operations). The key insight from the paper is that the replacement works; the missing piece for practical deployment is the position-shared architecture and a systematic distillation-to-deployment pipeline.

Knowledge distillation curricula for training deep sequence models. The paper's finding that feed-forward replacements can learn self-attention's function through distillation, but cannot (per the paper's assertion) discover it through end-to-end training, has implications for how deep Transformers are trained in low-resource settings. If attention provides an optimization scaffold that makes the translation function learnable from scratch, and if feed-forward networks have the capacity but not the "learnability," then a progressive distillation curriculum could be effective: first train a standard Transformer (where attention guides optimization to good solutions), then progressively replace attention blocks with distilled feed-forward networks, optionally fine-tuning the hybrid model end-to-end between replacement stages. This would be analogous to progressive network growing or layer-wise training, using attention as a temporary scaffold that is removed once the surrounding network has learned its representations. The paper's ALR M result (achieving near-baseline BLEU when all six encoder layers are replaced simultaneously) demonstrates that the replacement can work in parallel, but a sequential curriculum (replace one layer, fine-tune, replace the next) might allow smaller replacement networks to succeed by adapting the surrounding layers to the new attentionless sub-layer.

Interpretability analysis through architectural substitution. The component-replacement methodology can serve as a tool for understanding what specific attention heads compute in large pre-trained models. By training a separate shallow feed-forward network to mimic each attention head individually (similar to the ASLR approach but with per-head distillation targets), and then analyzing the learned weights of the feed-forward network (which are a single large matrix of input-to-output connections, unlike attention's factored Q-K-V structure), one might extract position-specific interaction patterns that are more interpretable than attention maps. The feed-forward network's first-layer weights directly encode which input positions influence which output positions—a position-wise interaction matrix that can be read off without the need for running attention on specific inputs. This could complement existing interpretability methods for attention (attention map visualization, probing classifiers) by providing a static, input-independent summary of what each head has learned to do, averaged over the training distribution.

Informing hardware-software co-design for Transformer inference. The paper's finding that feed-forward networks can replicate self-attention's function—but require 700× more parameters to do so—provides a concrete, quantitative justification for why specialized attention hardware accelerators (e.g., for sparse attention, FlashAttention's tiled implementation) are worth developing: attention achieves a given level of sequence-processing performance with dramatically fewer parameters, and this parameter efficiency translates to memory efficiency in deployment. Conversely, if future work can close the parameter-efficiency gap (through position-shared architectures, better training, or architectural innovations that combine feed-forward simplicity with attention-like inductive biases), the hardware implications would shift: feed-forward-dominant architectures are easier to accelerate (dense matrix multiplies are the most optimized operation in deep learning hardware) than attention's gather-scatter-multiply-softmax pipeline. The paper's quantification of the current efficiency gap provides a clear target for the hardware-architecture co-design community: reduce the 700× overhead, and the hardware argument flips.

---

When to Prefer This Method

The paper does not propose a method that practitioners should prefer over existing alternatives in any deployment scenario. The replacement networks are larger, less flexible, and restricted to fixed-length sequences. The authors are explicit that this is an analytical contribution rather than a practical one.

However, the paper does implicitly define boundary conditions where the approach of testing architectural necessity via distillation-based replacement could be productively applied by other researchers. These are not deployment preferences but methodological heuristics for using the paper's technique in architectural analysis:

• Use component-level distillation replacement when the goal is to test whether an architectural component provides representational value or optimization value. If a generic replacement (shallow MLP, 1D CNN) can match the component's function when trained via distillation but not when trained end-to-end, the component's contribution is primarily optimizational. If the replacement cannot match even with distillation, the component provides genuine representational value.

• Use graduated replacement levels (single component → component + surrounding structure → entire block) to localize which elements of a complex architecture are essential. The paper's ALR → ALRR → ELR hierarchy demonstrates how removing one structural element at a time separates their contributions.

• Test different attention types independently rather than treating attention as monolithic. The paper's separate experiments for encoder self-attention, decoder self-attention, and cross-attention reveal qualitatively different replaceability, which would be invisible in a whole-model replacement study. Any architectural analysis of the Transformer should similarly disaggregate attention types.