ArXiv: 1512.03385

🎯 Pitch

Adding more layers to a neural network can make it perform worseβ€”not just on test data, but even on the training set. This paper shows that making the layers learn residuals (the difference from the input) instead of the full mapping suddenly lets you train a 152-layer network that crushes shallower models, winning ImageNet with an ensemble error of just 3.57%.

1. Executive Summary

This paper introduces a deep residual learning framework that reformulates stacked neural network layers to learn residual functions (the difference between a desired underlying mapping and the layer input) rather than directly fitting unreferenced target functions, addressing the degradation problem where deeper networks paradoxically exhibit higher training error. Evaluated on ImageNet with residual networks (ResNets) up to 152 layersβ€”8Γ— deeper than VGG nets yet with lower computational complexityβ€”the approach enables accuracy gains from substantially increased depth where plain counterparts fail, with a 152-layer ResNet achieving 4.49% top-5 single-model validation error and an ensemble reaching 3.57% top-5 test error, winning ILSVRC 2015. On CIFAR-10, the framework successfully trains networks of 100 and 1000 layers, and on COCO object detection, replacing VGG-16 with ResNet-101 yields a 28% relative improvement in mAP@[.5, .95] solely from deeper learned representations, establishing that residual learning permits optimization of extremely deep networks only when layers are reformulated to fit residual functions with reference to identity shortcut connections.

2. Context and Motivation

The Core Problem: Deeper Networks Should Be Better, But They Aren't

The fundamental puzzle this paper addresses is deceptively simple: if you take a neural network that works well and make it deeper by stacking additional layers, why does it sometimes get worse? This is not just worse on held-out test data (which would indicate overfitting), but worse even on the training data itself β€” the very data used to optimize it. This phenomenon, which the paper terms the degradation problem, flies in the face of our basic intuition about model capacity.

To understand why this is so puzzling, consider a concrete example. Suppose you have a 20-layer network that achieves 5% training error on some task. Now you construct a 56-layer network by taking that same 20-layer architecture and appending 36 additional layers after it. At minimum, you could set those extra 36 layers to perform identity mapping β€” passing their input straight through to their output without modification β€” and copy the first 20 layers' weights from the trained shallower network. This constructed solution would achieve exactly the same 5% training error as the shallower network. Therefore, the solution space of the deeper network contains solutions that are at least as good as the best solution of the shallower network. Any halfway-competent optimizer should find a solution no worse than the shallower one.

Yet the paper's Figure 1 (left) shows the opposite: the 56-layer plain network on CIFAR-10 has higher training error than the 20-layer network. The same pattern appears on ImageNet in Figure 4 (left): the 34-layer plain network trains worse than the 18-layer one. This is not overfitting β€” it is a failure of optimization. The solver (SGD with backpropagation) simply cannot find weights that perform as well as the constructed identity-mapping solution, even though those weights definitively exist in the hypothesis space.

The paper explicitly frames this as a solver problem, not a model capacity problem:

"The degradation (of training accuracy) indicates that not all systems are similarly easy to optimize... The existence of this constructed solution indicates that a deeper model should produce no higher training error than its shallower counterpart. But experiments show that our current solvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time)."

This reframing is crucial: it tells us that the bottleneck isn't that deep networks are incapable of representing good functions, but rather that our optimization algorithms are incapable of finding those good functions within the available search space.

Why This Problem Matters: Depth Is Everything

The paper is motivated by a clear trend in computer vision at the time of writing (2015): depth of representations was the single most important architectural factor driving performance improvements. The evidence was overwhelming:

  • VGG nets (Simonyan and Zisserman, 2015) won the ImageNet challenge with 16–19 layers, explicitly demonstrating through controlled experiments that deeper configurations consistently outperformed shallower ones.
  • GoogLeNet (Szegedy et al., 2015) pushed to 22 layers (and later more) with the Inception architecture, achieving state-of-the-art results.
  • PReLU-net (He et al., 2015) reached 30 layers while maintaining trainability through carefully designed initialization.
  • Across visual recognition tasks β€” object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), and others β€” deeper representations extracted from ImageNet-pretrained models consistently improved downstream performance.

The paper states this motivation directly:

"Deep networks naturally integrate low/mid/high-level features and classifiers in an end-to-end multi-layer fashion, and the 'levels' of features can be enriched by the number of stacked layers (depth). Recent evidence reveals that network depth is of crucial importance."

The implication is clear: if depth is the primary lever for better representations, and optimization difficulties prevent us from exploiting depth beyond a certain point, then solving the optimization bottleneck would unlock a new regime of representational power. Every additional layer that could be successfully trained would translate directly into better features, better classifications, and better performance on downstream tasks. The practical stakes were enormous β€” whoever solved depth scaling would dominate the major vision benchmarks.

At the same time, there is a deep theoretical question lurking underneath: why can't SGD find identity-like solutions? Neural networks with ReLU activations can represent identity mappings (a single layer with weight matrix equal to the identity matrix and zero bias does exactly this). Batch normalization (Ioffe and Szegedy, 2015) had addressed the vanishing/exploding gradient problem that previously prevented deep networks from even starting to converge, meaning gradients were flowing. Yet despite having both the representational capacity and the gradient signal, the optimizer was failing. Understanding why would reveal something fundamental about the optimization landscape of deep networks.

Prior Approaches and Where They Fall Short

The paper situates its contribution against two categories of prior work: attempts to address the specific degradation problem, and general-purpose techniques that had enabled depth up to a point but hit limits.

Vanishing/Exploding Gradients (The Solved Problem)

The most famous historical obstacle to deep network training was vanishing or exploding gradients (Bengio et al., 1994; Glorot and Bengio, 2010). When gradients propagated backward through many layers, they could either shrink exponentially (vanishing) or grow exponentially (exploding), making effective weight updates in early layers essentially impossible.

This problem had been largely addressed by the time of the ResNet paper through two key innovations:

  • Normalized initialization schemes (LeCun et al., 1998; Glorot and Bengio, 2010; He et al., 2015) that carefully set the scale of initial weights to preserve gradient magnitudes through the network.
  • Batch normalization (BN; Ioffe and Szegedy, 2015), which normalizes activations within each mini-batch, ensuring that forward-propagated signals maintain non-zero variance and backward-propagated gradients exhibit healthy norms regardless of depth.

The paper explicitly argues that the degradation problem is not caused by vanishing gradients:

"We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN, which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish."

This is a critical distinction. The degradation problem persists after vanishing gradients are solved, meaning it represents a qualitatively different optimization difficulty. The 34-layer plain network in Figure 4 (left) still converges to some extent β€” it achieves competitive accuracy β€” but it consistently underperforms the shallower 18-layer variant, and extended training (3Γ— iterations, per the paper's footnote) doesn't close the gap. The paper speculates that "deep plain nets may have exponentially low convergence rates," implying that the optimization landscape becomes progressively more ill-conditioned with depth, even though gradient magnitudes remain healthy.

Highway Networks (The Closest Prior Work)

Concurrent with this work, Srivastava et al. (2015) introduced highway networks, which share the idea of shortcut connections but differ in a fundamental way. Highway networks use gated shortcuts: the network learns, via sigmoid gating functions with their own parameters, how much of the input to pass through (the "transform gate") versus how much to transform (the "carry gate"). These gates are data-dependent and have learnable parameters.

The paper draws a sharp contrast with ResNets on several axes:

"These gates are data-dependent and have parameters, in contrast to our identity shortcuts that are parameter-free. When a gated shortcut is 'closed' (approaching zero), the layers in highway networks represent non-residual functions. On the contrary, our formulation always learns residual functions; our identity shortcuts are never closed, and all information is always passed through, with additional residual functions to be learned."

In other words, highway networks allow the network to learn an identity mapping if the gate parameters converge to that behavior, but they don't bias the optimization toward identity. The network could just as easily close the gate and behave like a plain network. ResNets, by contrast, hardwire the identity mapping as the default behavior and learn only the residual β€” the deviation from identity. This is a stronger inductive bias: the network starts at identity and learns perturbations, rather than starting from scratch and optionally learning to pass information through.

Crucially, the paper notes that highway networks "have not demonstrated accuracy gains with extremely increased depth (e.g., over 100 layers)." This suggests that the gating mechanism, while related in spirit, does not solve the optimization problem as effectively as hardwired identity shortcuts. The ResNet's design choice to always pass information through and learn only residuals turns out to be the key that enables 152-layer (and even 1000-layer) training.

Other Shortcut Connection Variants

The paper acknowledges several precursors to the idea of skip connections, but each has limitations:

  • Early MLP training (Ripley, 1996; Venables and Ripley, 1999) sometimes added a linear layer directly from network input to output. This is a single global shortcut β€” not the pervasive, every-few-layers pattern that ResNets use.
  • Auxiliary classifiers attached to intermediate layers (Szegedy et al., 2015; Lee et al., 2014) inject gradient signals at those points, helping with vanishing gradients. But these don't change the layer-to-layer computation β€” they just add supervision signals β€” and don't address the degradation problem after gradients are already healthy.
  • Centering techniques (Schraudolph, 1998a, 1998b; Raiko et al., 2012; Vatanen et al., 2013) reparameterize layers to have zero-mean responses or gradients, which improves conditioning but doesn't address the core identity-approximation difficulty.
  • Inception layers (Szegedy et al., 2015) include a shortcut branch in parallel with deeper branches, but this is an architectural motif for multi-scale feature extraction rather than a systematic approach to enabling depth.

None of these approaches explicitly reformulate the learning objective in terms of residuals with reference to identity, which the paper argues is the essential insight.

How This Paper Positions Itself

The paper does not claim that the idea of residual representations or shortcut connections is entirely new. It explicitly acknowledges precursors in both neural network history and classical signal processing:

  • In image recognition, residual vectors with respect to a dictionary (VLAD; JΓ©gou et al., 2012) and Fisher Vectors (Perronnin and Dance, 2007) encode information as deviations from reference points, and encoding residuals has been shown more effective than encoding original vectors in vector quantization (JΓ©gou et al., 2011).
  • In scientific computing, the Multigrid method (Briggs et al., 2000) and hierarchical basis preconditioning (Szeliski, 1990, 2006) reformulate PDE systems to solve for residual solutions at multiple scales, converging much faster than standard solvers. The paper draws an explicit analogy: "These methods suggest that a good reformulation or preconditioning can simplify the optimization."

The paper's self-positioning is thus: the specific reformulation of deep network layers to learn residual functions with reference to identity, implemented via pervasive parameter-free identity shortcut connections, is a simple yet transformative idea that addresses the degradation problem in a way that prior neural network techniques cannot. The key differentiators are:

  1. Hardwired identity, not learned gating. Unlike highway networks, the identity path is always open. The network's default behavior is to copy its input; learning only needs to discover useful modifications to that copy. This makes optimization dramatically easier.

  2. Universality and simplicity. The building block (Figure 2) can be inserted into any existing architecture with essentially no additional parameters or computation (only element-wise addition). It doesn't require new solvers, new initialization schemes, or new hardware β€” existing SGD with backpropagation works out of the box.

  3. Empirical demonstration of extreme depth. Prior methods had enabled training of networks with tens of layers (VGG-19, GoogLeNet-22, PReLU-net-30). This paper demonstrates successful training at 152 layers on ImageNet and 1202 layers on CIFAR-10 β€” scales that were simply not achievable before. The paper is making a statement not just about a technique but about a new depth regime that the technique unlocks.

  4. The degradation problem as a first-class phenomenon. Prior work had observed that very deep networks sometimes performed worse (He and Sun, 2015; Srivastava et al., 2015), but the paper provides the most systematic characterization (Figures 1, 4, 6 left panels) and explicitly frames it as the central challenge that residual learning solves. This framing β€” the degradation problem as a solver failure rather than a capacity problem β€” is itself a conceptual contribution that shapes the entire approach.

The paper's ambition extends beyond classification. By showing that ResNet-101 improves COCO object detection mAP by 28% relative solely through better learned representations (Tables 7–8), and that these representations underpin winning entries across five competition tracks (ImageNet classification, detection, localization; COCO detection, segmentation), the paper positions residual learning as a general principle for representation learning rather than a classification-specific trick. The concluding statement of the introduction captures this: "This strong evidence shows that the residual learning principle is generic, and we expect that it is applicable in other vision and non-vision problems."

3. Technical Approach

3.1 Reader Orientation

This is primarily a architectural innovation paper whose core idea is that neural network layers should learn residual functions β€” the difference between a desired output and the input β€” rather than directly learning the desired output itself.

The system being built is a convolutional neural network that can be made arbitrarily deep (hundreds or thousands of layers) without suffering from the degradation problem β€” the counterintuitive phenomenon where deeper networks produce higher training error than shallower ones. The "shape" of the solution is deceptively simple: for every few stacked layers in the network, add a skip connection that passes the input straight through to the output via identity mapping, and train those layers to learn only the residual correction to that identity path.

3.2 Big-Picture Architecture (Diagram in Words)

The ResNet architecture has three conceptual components:

  1. Plain convolutional backbone: A standard CNN with convolutional layers organized into stages of decreasing spatial resolution and increasing channel count, following VGG-style design rules (3Γ—3 convolutions, doubling filters when halving feature map size).

  2. Identity shortcut connections: Parameter-free pathways that bypass every two or three convolutional layers, adding the block's input directly to its output via element-wise addition. These are the defining architectural feature that distinguishes ResNets from plain networks.

  3. Bottleneck blocks (for deeper variants): A three-layer design replacing the basic two-layer block, using 1Γ—1 convolutions to first compress and then restore channel dimensions, with a 3Γ—3 convolution operating on the compressed representation in between. This maintains computational tractability at extreme depths.

Information flows through the network as follows: an image enters the initial convolutional stem (7Γ—7 conv, stride 2, followed by max pooling) β†’ passes through a series of residual blocks organized into four stages, each stage operating at a specific spatial resolution β†’ within each block, the signal splits into two paths: the residual path (two or three convolutions with BN and ReLU) and the identity path (direct connection, possibly with 1Γ—1 projection if dimensions change) β†’ the two paths are summed element-wise β†’ a final ReLU is applied β†’ the summed signal becomes the input to the next block β†’ after all stages, global average pooling collapses spatial dimensions β†’ a fully-connected layer produces class predictions.

3.3 Roadmap for the Deep Dive

  • First, the mathematical formulation of residual learning (the core equation $y = \mathcal{F}(x, \{W_i\}) + x$), because this single equation encodes the entire insight: what changes versus a plain network, why the identity shortcut is the crucial inductive bias, and how it preconditions the optimization landscape.

  • Second, the building block designs (basic two-layer block and bottleneck three-layer block), including how they handle dimension mismatches via zero-padding or 1Γ—1 projections, because these are the concrete instantiations of the residual principle.

  • Third, the full network architectures for ImageNet (18, 34, 50, 101, 152 layers) and CIFAR-10 (20 to 1202 layers), mapping the logical principle to specific layer configurations, filter counts, and downsampling strategies, since the architectural details determine the actual depth scaling behavior.

  • Fourth, the training methodology (SGD hyperparameters, data augmentation, weight initialization, learning rate schedules), because the paper's central claim is that residual networks are easier to optimize β€” this claim only makes sense in the context of specific optimization procedures.

  • Fifth, the design rationale for key choices (why identity shortcuts over gated shortcuts, why bottleneck over basic blocks, why projection shortcuts are not essential), connecting each decision back to the core degradation problem and the empirical evidence.

3.4 Detailed, Sentence-Based Technical Breakdown

The Residual Learning Formulation

The paper's foundational idea is expressed in a single equation that redefines what a stack of neural network layers should compute. In a traditional "plain" network, each stack of layers (say, two convolutional layers) is tasked with directly learning some desired underlying mapping $\mathcal{H}(x)$ β€” given an input $x$, produce the correct output. The residual learning reformulation changes this: instead of learning $\mathcal{H}(x)$ directly, the layers learn a residual function $\mathcal{F}(x)$ defined as the difference between the desired output and the input:

F(x):=H(x)βˆ’x\mathcal{F}(x) := \mathcal{H}(x) - x

where $\mathcal{H}(x)$ is the desired underlying mapping that the stack of layers should ultimately compute, $x$ is the input to the first layer in the stack, and $\mathcal{F}(x)$ is the residual β€” the correction that must be added to $x$ to obtain the correct output.

The original mapping is then recovered by adding the input back:

H(x)=F(x)+x\mathcal{H}(x) = \mathcal{F}(x) + x

What it computes: Given an input vector $x$ (e.g., a feature map tensor), the stacked layers produce a residual correction $\mathcal{F}(x)$ that captures how the desired output differs from a simple identity transformation. The actual output returned to the next part of the network is the sum of this residual and the original input. Operationally, this means the layers are no longer responsible for generating the entire output from scratch β€” they only need to model the deviation from passing the input through unchanged.

Why this form: The key insight is about the ease of optimization, not representational capacity. Both formulations β€” learning $\mathcal{H}(x)$ directly and learning $\mathcal{F}(x) = \mathcal{H}(x) - x$ β€” can asymptotically approximate the same set of functions (assuming universal approximation properties of neural networks). The difference is in what happens when the optimal function is close to identity. Consider the degradation problem: if adding more layers to a network should not hurt performance, then the optimal behavior for those added layers is identity mapping β€” they should pass their input through unchanged. In the plain formulation, the layers must learn to approximate an identity function $\mathcal{H}(x) = x$ using multiple nonlinear transformations (convolutions, ReLUs, batch normalizations). This turns out to be difficult for SGD β€” the paper hypothesizes that "the solvers might have difficulties in approximating identity mappings by multiple nonlinear layers." In the residual formulation, the layers need only learn $\mathcal{F}(x) = 0$ β€” pushing all weights toward zero, which is a much easier optimization target because weight decay already biases weights toward zero and because zero-initialized or near-zero weights produce small outputs by default. The residual formulation preconditions the optimization problem by making the default behavior (small random weights producing near-zero residuals) correspond to near-identity mapping, so the network starts close to a good solution and only needs to learn perturbations.

The paper explicitly draws an analogy to classical preconditioning in numerical optimization:

"If the optimal function is closer to an identity mapping than to a zero mapping, it should be easier for the solver to find the perturbations with reference to an identity mapping, than to learn the function as a new one."

In other words, residual learning converts the problem from "find the correct function in a vast space" to "find a small correction to the identity function," which is an easier search problem because the initialization already provides a reasonable starting point (near-identity behavior). This connects to the Multigrid and hierarchical basis preconditioning methods cited in the Related Work β€” those methods also reformulate problems in terms of residuals to accelerate convergence.

A subtle point: the paper does not claim that identity mappings are actually optimal in practice. The claim is more nuanced: "In real cases, it is unlikely that identity mappings are optimal, but our reformulation may help to precondition the problem." Even when the optimal function is far from identity, providing the identity as a reference point makes the learning problem better-conditioned because the solver can focus on learning the difference from a known starting point rather than synthesizing the entire function from scratch.

Identity Mapping by Shortcuts

The mathematical formulation $\mathcal{F}(x) + x$ is realized concretely through shortcut connections β€” pathways that bypass one or more layers and feed the input directly to a later point in the network. The paper's implementation uses shortcuts that skip every few stacked layers and perform a simple operation: element-wise addition of the shortcut signal and the output of the skipped layers.

The basic building block equation. Formally, a residual building block is defined as:

y=F(x,{Wi})+xy = \mathcal{F}(x, \{W_i\}) + x

where $x$ is the input tensor to the block (the output of the previous block or the initial stem), $\{W_i\}$ are the weight matrices (convolutional filters) of the layers inside the block, $\mathcal{F}(x, \{W_i\})$ is the residual mapping learned by those layers β€” what the block computes before adding the shortcut β€” and $y$ is the block's output, which becomes the input to the next block.

For the concrete two-layer block shown in Figure 2, the residual function expands to:

F=W2Οƒ(W1x)\mathcal{F} = W_2 \sigma(W_1 x)

where $W_1$ is the weight matrix (convolutional filters) of the first layer, $\sigma$ is the ReLU activation function applied after the first layer's batch normalization, and $W_2$ is the weight matrix of the second layer. Biases are omitted from the notation for simplicity but are present in the actual implementation.

What it computes (operational flow in a two-layer residual block): The input $x$ enters the block and simultaneously flows down two paths. On the residual path: $x$ passes through the first convolutional layer (producing $W_1 x$), then batch normalization, then ReLU (yielding $\sigma(W_1 x)$), then the second convolutional layer (producing $W_2 \sigma(W_1 x)$), then batch normalization. This produces the residual $\mathcal{F}(x)$. On the identity path: $x$ passes through unchanged β€” literally a wire. The two paths converge at an element-wise addition operation: $\mathcal{F}(x) + x$. After this addition, a final ReLU is applied: $\sigma(\mathcal{F}(x) + x)$. This ReLU-after-addition is important β€” it means the nonlinearity sees the combined signal, not just the residual.

Why this form (identity shortcuts, not parameterized transformations): The shortcut in Equation (1) is the identity function $x \mapsto x$. This choice has several critical properties:

  1. Parameter-free: The identity shortcut introduces exactly zero additional parameters. This means residual networks can be compared fairly to plain networks with the same number of parameters, depth, width, and computational cost (the element-wise addition is negligible). If the shortcut used a learned transformation (e.g., a 1Γ—1 convolution to project $x$ to a different space), the comparison would be confounded by the extra parameters.

  2. Always open: Unlike highway networks where a learned gate can close the shortcut (approaching zero throughput), the identity shortcut is always passing information through at full strength. The network cannot learn to ignore the shortcut and behave like a plain network β€” it is structurally forced to learn residuals. This hard constraint is what the paper argues makes optimization easier, because the optimizer cannot fall into the trap of trying to learn the entire function from scratch when that proves difficult.

  3. Computationally trivial: Element-wise addition of two tensors of the same shape costs almost nothing compared to convolution operations (which involve many multiply-adds per output element). This means the residual mechanism adds essentially no overhead to training or inference.

Handling dimension mismatches. The identity shortcut $x$ can only be directly added to $\mathcal{F}(x)$ if they have the same dimensions β€” same number of channels (feature maps), same spatial height and width. When the residual path changes dimensions (e.g., after a strided convolution that halves spatial resolution and doubles the number of channels), the shortcut must adapt. The paper proposes two options:

y=F(x,{Wi})+Wsxy = \mathcal{F}(x, \{W_i\}) + W_s x

where $W_s$ is a linear projection matrix (implemented as a 1Γ—1 convolution) used only when the dimensions of $x$ and $\mathcal{F}$ differ.

Option A β€” Zero-padding: The shortcut still performs identity mapping, but the missing dimensions are filled with zeros. For example, if $\mathcal{F}(x)$ has twice as many channels as $x$, half the channels in the shortcut output are simply padded with zeros. This option introduces no extra parameters β€” the element-wise addition still works because zero-padding makes the tensor shapes match. The zero-padded channels receive no residual information from the shortcut, so the residual path must learn those channels entirely on its own.

Option B β€” Projection shortcuts: A 1Γ—1 convolution is applied to $x$ to match the number of channels and spatial stride of $\mathcal{F}(x)$. When the shortcuts go across feature maps of two spatial sizes (e.g., in the downsampling blocks), they are performed with a stride of 2. This introduces a small number of additional parameters β€” one convolution per dimension-changing shortcut.

Option C β€” All shortcuts are projections: Every shortcut connection, even when dimensions match, uses a learned projection. This is the most parameter-heavy option and is only evaluated to isolate the effect of projection parameters.

The empirical finding (Table 3) is that all three options are substantially better than the plain network, with small differences among them: B slightly outperforms A (because zero-padded dimensions in A have no residual learning), and C marginally outperforms B (due to extra parameters). The crucial conclusion: projection shortcuts are not essential for addressing the degradation problem. Identity shortcuts with zero-padding for dimension matching (option A) already solve the degradation problem, confirming that the residual formulation itself β€” not additional parameters β€” is what enables deep training. The paper uses option B for deeper networks (50/101/152 layers) as a minor accuracy improvement, and option A for the CIFAR-10 experiments.

Building Block Designs

The residual function $\mathcal{F}$ is flexible β€” it can contain any number of layers, any types of operations, and any connectivity pattern inside the block. The paper explores two specific block designs, chosen to balance representational power with computational efficiency.

Basic two-layer block (Figure 5, left; used in ResNet-18 and ResNet-34). This block contains two 3Γ—3 convolutional layers, each followed by batch normalization and ReLU (though the ReLU after the first conv is applied before the second conv, and the ReLU after the second conv is delayed until after the addition with the shortcut). The sequence inside the residual path is:

  1. 3Γ—3 convolution β†’ Batch Normalization β†’ ReLU
  2. 3Γ—3 convolution β†’ Batch Normalization
  3. Element-wise addition with shortcut
  4. ReLU

The two 3Γ—3 convolutions each have the same number of input and output channels within a block (e.g., two 64-channel 3Γ—3 convolutions in the early stages). This means the spatial receptive field of the residual function is 5Γ—5 (two 3Γ—3 layers stacked), giving the block a modest capacity to model spatial interactions.

For ResNet-34 on ImageNet, this block is used with the following configuration: the first stage (conv2_x) has 3 blocks with 64-channel 3Γ—3 convolutions operating on 56Γ—56 feature maps; the second stage (conv3_x) has 4 blocks with 128-channel convolutions on 28Γ—28 maps; the third stage (conv4_x) has 6 blocks with 256-channel convolutions on 14Γ—14 maps; the fourth stage (conv5_x) has 3 blocks with 512-channel convolutions on 7Γ—7 maps. Downsampling occurs at the first block of stages 2–4 via a stride-2 convolution in the first 3Γ—3 layer.

Bottleneck three-layer block (Figure 5, right; used in ResNet-50, ResNet-101, ResNet-152). For networks deeper than 34 layers, the basic two-layer block becomes computationally expensive because every 3Γ—3 convolution operates on the full channel dimension (e.g., two 256-channel 3Γ—3 convolutions in conv4_x, each requiring $256 \times 256 \times 3 \times 3$ parameters). The bottleneck design reduces this cost by sandwiching a single 3Γ—3 convolution between two 1Γ—1 convolutions that first reduce and then restore the channel dimension:

  1. 1Γ—1 convolution (reduces channels, e.g., 256 β†’ 64) β†’ Batch Normalization β†’ ReLU
  2. 3Γ—3 convolution (operates on reduced channels, e.g., 64 β†’ 64) β†’ Batch Normalization β†’ ReLU
  3. 1Γ—1 convolution (restores channels, e.g., 64 β†’ 256) β†’ Batch Normalization
  4. Element-wise addition with shortcut
  5. ReLU

The first 1Γ—1 convolution compresses the representation into a lower-dimensional "bottleneck," the 3Γ—3 convolution performs spatial processing in this compressed space, and the second 1Γ—1 convolution expands back to the original dimension. The paper's example (Figure 5) shows a 256-dimensional input being reduced to 64 dimensions, processed, and restored to 256 dimensions.

Why the bottleneck design: The computational savings are substantial. A two-layer basic block with 256 input/output channels performs two 3Γ—3 convolutions, each costing $256 \times 256 \times 3 \times 3 = 589,824$ multiply-adds per spatial position, totaling ~1.18 million. A bottleneck block with reduction factor 4 performs: a 1Γ—1 convolution (256 β†’ 64: $256 \times 64 = 16,384$), a 3Γ—3 convolution (64 β†’ 64: $64 \times 64 \times 3 \times 3 = 36,864$), and a 1Γ—1 convolution (64 β†’ 256: $64 \times 256 = 16,384$), totaling ~69,632 multiply-adds per spatial position β€” roughly 17Γ— fewer operations. This makes 50, 101, and 152-layer networks computationally feasible while still providing more parameters and nonlinearities than shallower basic-block networks.

The parameter-free identity shortcut is "particularly important for the bottleneck architectures" because the two ends of the block (input and output) are high-dimensional (e.g., 256 channels), while the interior is low-dimensional (64 channels). If the shortcut were replaced with a projection, it would need to connect the high-dimensional input to the high-dimensional output β€” a 256 β†’ 256 1Γ—1 convolution β€” which has $256 \times 256 = 65,536$ parameters, roughly doubling the block's total parameter count and computational cost. Identity shortcuts avoid this penalty entirely.

The paper notes a practical constraint: deeper non-bottleneck ResNets (like a hypothetical 101-layer basic-block network) "also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations." The bottleneck is thus an engineering optimization for computational efficiency, not a fundamental requirement for residual learning β€” the CIFAR-10 experiments with 110 and 1202 layers use basic two-layer blocks throughout, confirming that residual learning works without bottlenecks.

Single-layer residual block (a degenerate case): The paper notes that if $\mathcal{F}$ contains only a single layer, Equation (1) reduces to $y = W_1 x + x$, which is essentially a linear layer with an identity skip connection. The authors report they "have not observed advantages" for this configuration. This suggests that the residual formulation requires at least two layers of nonlinear processing to be effective β€” a single linear layer plus skip connection does not provide enough representational capacity to benefit from the residual reformulation.

Full Network Architectures

The paper provides complete architectural specifications for two dataset scales: ImageNet (large-scale classification, 1.28M training images, 1000 classes) and CIFAR-10 (small-scale, 50k training images, 10 classes). The architectures share the same residual learning principle but differ in depth, width, and input processing to match their respective problem scales.

ImageNet Plain Network (Figure 3, middle; Table 1). The baseline architecture follows "the philosophy of VGG nets" with two design rules:

  1. Rule 1: For layers operating on feature maps of the same spatial size (i.e., within the same processing stage), all convolutional layers have the same number of output channels (filters). This creates uniform representational capacity within each stage.

  2. Rule 2: When the spatial resolution is halved (via strided convolution), the number of filters is doubled. This "preserves the time complexity per layer" β€” roughly, each layer does the same total amount of computation because halving height and width (ΒΌ the activations) is compensated by doubling channels (4Γ— the computation per position).

The network structure:

  • Stem (conv1): A 7Γ—7 convolution with 64 filters, stride 2 (reducing 224Γ—224 input to 112Γ—112), followed by 3Γ—3 max pooling with stride 2 (reducing to 56Γ—56). This initial downsampling reduces spatial resolution aggressively to keep computation manageable.

  • Stage 1 (conv2_x): Operates on 56Γ—56 feature maps. Contains $x$ pairs of 3Γ—3 convolutions, all with 64 filters. For the 18-layer net, $x = 2$ (4 conv layers total); for the 34-layer net, $x = 3$ (6 conv layers).

  • Stage 2 (conv3_x): Operates on 28Γ—28 feature maps (downsampled from 56Γ—56 by a stride-2 3Γ—3 convolution in the first layer). Contains $x$ pairs of 3Γ—3 convolutions with 128 filters. For 18-layer: $x = 2$ (4 conv layers); for 34-layer: $x = 4$ (8 conv layers).

  • Stage 3 (conv4_x): Operates on 14Γ—14 feature maps (downsampled from 28Γ—28). Contains $x$ pairs of 3Γ—3 convolutions with 256 filters. For 18-layer: $x = 2$ (4 conv layers); for 34-layer: $x = 6$ (12 conv layers).

  • Stage 4 (conv5_x): Operates on 7Γ—7 feature maps (downsampled from 14Γ—14). Contains $x$ pairs of 3Γ—3 convolutions with 512 filters. For 18-layer: $x = 2$ (4 conv layers); for 34-layer: $x = 3$ (6 conv layers).

  • Classifier head: Global average pooling collapses the 7Γ—7 spatial dimensions to 1Γ—1, producing a 512-dimensional vector. A 1000-way fully-connected layer with softmax produces class probabilities.

The total number of weighted layers (convolutional and fully-connected) is counted by summing all layers with learnable parameters. The 34-layer plain network has: 1 (conv1) + 6 (conv2_x) + 8 (conv3_x) + 12 (conv4_x) + 6 (conv5_x) + 1 (fc) = 34 layers. Its computational cost is 3.6 billion FLOPs (multiply-add operations), compared to 19.6 billion for VGG-19 β€” only 18% of VGG-19's cost despite having nearly twice the depth (34 vs. 19 layers). This efficiency comes from the aggressive early downsampling (7Γ—7 conv with stride 2, then max pool with stride 2) and the absence of the large fully-connected layers (4096-d fc layers) that dominate VGG's parameter count and computation.

ImageNet Residual Network (Figure 3, right; Table 1). The residual version is constructed by taking the plain network architecture and inserting shortcut connections between every pair (or triplet, for bottleneck) of 3Γ—3 convolutional layers. The layers themselves, their filter counts, and their arrangement are identical to the plain counterpart β€” only the shortcuts are added. This means residual and plain networks can be compared at exactly equal parameter counts.

The transformation from plain to residual:

  • Each pair of 3Γ—3 convolutions in the plain network becomes a residual block: the output of the second convolution is added element-wise to the input of the first convolution, followed by ReLU.
  • Within a stage where the number of channels doesn't change (e.g., conv2_x where all convolutions have 64 filters), the shortcut is a bare identity: the input tensor is added directly to the output without any transformation. The paper calls these "solid line shortcuts" in Figure 3.
  • At the first block of stages 2, 3, and 4, the spatial resolution and channel count both change (e.g., conv3_x's first block goes from 64 channels at 56Γ—56 to 128 channels at 28Γ—28). The shortcut must adapt: either zero-padding (option A) or a 1Γ—1 convolution with stride 2 (option B). The paper calls these "dotted line shortcuts" in Figure 3.

ResNet-18 and ResNet-34 use the basic two-layer block throughout. ResNet-34 has: 1 (conv1) + 3Γ—2 (conv2_x, three blocks of two conv layers each) + 4Γ—2 (conv3_x) + 6Γ—2 (conv4_x) + 3Γ—2 (conv5_x) + 1 (fc) = 34 layers. The computational cost remains 3.6 billion FLOPs β€” exactly the same as the plain 34-layer network because the identity shortcuts add zero computation beyond the negligible element-wise addition.

ResNet-50, ResNet-101, and ResNet-152 use the bottleneck three-layer block. Each block now contains three convolutional layers (1Γ—1, 3Γ—3, 1Γ—1) instead of two, so the layer count per stage increases: three times the number of blocks per stage. The specific configurations from Table 1:

  • ResNet-50: conv2_x has 3 bottleneck blocks (9 conv layers), conv3_x has 4 blocks (12 layers), conv4_x has 6 blocks (18 layers), conv5_x has 3 blocks (9 layers). Total: 1 (conv1) + 9 + 12 + 18 + 9 + 1 (fc) = 50 layers. FLOPs: 3.8 billion.
  • ResNet-101: conv2_x has 3 blocks, conv3_x has 4 blocks, conv4_x has 23 blocks (69 conv layers!), conv5_x has 3 blocks. Total: 1 + 9 + 12 + 69 + 9 + 1 = 101 layers. FLOPs: 7.6 billion.
  • ResNet-152: conv2_x has 3 blocks, conv3_x has 8 blocks, conv4_x has 36 blocks (108 conv layers!), conv5_x has 3 blocks. Total: 1 + 9 + 24 + 108 + 9 + 1 = 152 layers. FLOPs: 11.3 billion.

The bottleneck block's internal channel dimensions follow a consistent pattern: for a block with $C$ output channels, the intermediate (bottleneck) channels are $C/4$. For example, in conv4_x where output channels are 1024, the bottleneck operates at 256 channels. The expansion factor of 4 is chosen empirically β€” the paper uses it without extensive ablation, suggesting it was found to work well in practice.

Notably, the 152-layer ResNet (11.3 billion FLOPs) still has lower computational complexity than VGG-16 (15.3 billion FLOPs) and VGG-19 (19.6 billion FLOPs), despite being roughly 8Γ— deeper. This efficiency is achieved through: (1) no large fully-connected layers (VGG's three 4096-d fc layers contain ~120M of VGG-16's 138M parameters), (2) aggressive early downsampling (the 7Γ—7 conv with stride 2 and initial max pool), and (3) the bottleneck design.

CIFAR-10 Architecture. For the smaller 32Γ—32 CIFAR-10 images, the paper uses a simpler architecture designed to test the behavior of extremely deep networks rather than push state-of-the-art accuracy. The structure follows the same pattern as the ImageNet plain/residual nets but scaled down:

  • The input is a 32Γ—32 image with per-pixel mean subtraction.
  • The first layer is a single 3Γ—3 convolution (no 7Γ—7 conv, no max pool β€” the spatial resolution is already small).
  • Three stages of processing operate on feature maps of sizes 32Γ—32, 16Γ—16, and 8Γ—8 respectively. Each stage contains $2n$ convolutional layers (grouped into $n$ residual blocks of two layers each, with shortcut connections between each pair).
  • Filter counts are 16, 32, and 64 for the three stages β€” substantially smaller than ImageNet because CIFAR-10 has only 10 classes and 50k training images, so fewer parameters are needed.
  • Subsampling from one stage to the next is performed by a stride-2 convolution.
  • The network ends with global average pooling, a 10-way fully-connected layer, and softmax.
  • The total number of stacked weighted layers is $6n + 2$ (1 initial conv + $2n$ layers per stage across 3 stages + 1 fc layer).

The paper sweeps $n = \{3, 5, 7, 9\}$ to produce 20, 32, 44, and 56-layer networks, with $n = 18$ for a 110-layer network and $n = 200$ for a 1202-layer network. All shortcuts use identity mapping with zero-padding for dimension increases (option A), meaning the residual networks have exactly the same number of parameters as their plain counterparts.

The parameter counts from Table 6 show the scale: the 20-layer ResNet has only 0.27M parameters, the 110-layer ResNet has 1.7M, and the 1202-layer ResNet has 19.4M. The 1202-layer model's relatively small parameter count (for its depth) comes from the narrow architecture β€” only 16/32/64 filters per stage β€” and the absence of large fully-connected layers.

Training Methodology

The paper's training procedures are standard for the time (following Krizhevsky et al., 2012 and Simonyan and Zisserman, 2015) but are worth documenting precisely because the central claim β€” residual networks are easier to optimize β€” is evaluated under these specific optimization conditions.

ImageNet Training (Section 3.4).

  • Data preprocessing: Images are resized with the shorter side randomly sampled from $[256, 480]$ for scale augmentation (following VGG). A 224Γ—224 crop is randomly sampled from the resized image or its horizontal flip. The per-pixel mean (computed over the training set) is subtracted. Standard color augmentation from Krizhevsky et al. (2012) is applied.
  • Batch normalization: BN is applied "right after each convolution and before activation" β€” meaning the layer order is: convolution β†’ BN β†’ ReLU. This follows Ioffe and Szegedy (2015).
  • Weight initialization: Weights are initialized following He et al. (2015) β€” the MSRA initialization designed for ReLU activations, which draws weights from a zero-mean Gaussian with variance $2/n_{\text{in}}$ where $n_{\text{in}}$ is the number of input units (fan-in). This initialization accounts for the ReLU's halving of variance.
  • Optimizer: Stochastic gradient descent (SGD) with mini-batch size 256.
  • Learning rate schedule: Learning rate starts at 0.1 and is divided by 10 when the error plateaus. The models are trained for "up to $60 \times 10^4$ iterations" (600k iterations). The specific plateau detection criterion is not detailed, but this is standard practice β€” monitor validation error and drop the learning rate when improvement stalls.
  • Regularization: Weight decay of 0.0001, momentum of 0.9. No dropout is used, "following the practice in [16]" (Ioffe and Szegedy, 2015, who showed BN reduces the need for dropout).
  • Training from scratch: All networks are trained from random initialization (no pretraining).

ImageNet Testing (Section 3.4).

  • Comparison studies: Standard 10-crop testing (Krizhevsky et al., 2012) β€” 10 fixed crops (center + 4 corners Γ— horizontal flip) of 224Γ—224 are evaluated, and their softmax probabilities are averaged.
  • Best results: The fully-convolutional form (Simonyan and Zisserman, 2015; He et al., 2015) is used β€” the network is applied as a fully-convolutional network on the input image without cropping to fixed 224Γ—224 patches, producing a spatial map of class scores that is averaged. This is done at multiple scales: the shorter side of the image is resized to $\{224, 256, 384, 480, 640\}$, and scores are averaged across all scales.

CIFAR-10 Training.

  • Data preprocessing: Inputs are 32Γ—32 images with per-pixel mean subtraction. Data augmentation follows Lee et al. (2014): 4 pixels are padded on each side (producing a 40Γ—40 image), and a random 32Γ—32 crop is sampled from the padded image or its horizontal flip.
  • Batch normalization and weight initialization: Same as ImageNet (BN after each convolution, MSRA initialization).
  • Optimizer: SGD with mini-batch size 128 on two GPUs.
  • Learning rate schedule: Learning rate starts at 0.1, divided by 10 at 32k and 48k iterations, training terminates at 64k iterations. This schedule was "determined on a 45k/5k train/val split" β€” meaning the 50k training images were split into 45k for training and 5k for validation to tune the schedule, after which the full 50k training set was used for the final models.
  • Regularization: Weight decay of 0.0001, momentum of 0.9. No dropout.
  • Training from scratch: All models trained from random initialization.
  • Testing: Only a single view of the original 32Γ—32 image is evaluated (no 10-crop or multi-scale testing).

CIFAR-10 110-layer learning rate warmup. For the 110-layer ResNet, the initial learning rate of 0.1 was "slightly too large to start converging" β€” the model would diverge initially before eventually finding its way. The solution: use a learning rate of 0.01 for a "warmup" period until the training error drops below 80% (about 400 iterations), then switch to 0.1 and continue with the normal schedule. This is an early, informal version of the learning rate warmup that later became standard practice for training very deep networks. The paper notes that with 0.1 from the start, the network "starts converging (<90% error) after several epochs, but still reaches similar accuracy" β€” the warmup simply stabilizes early training.

Why these training choices matter for the paper's claims. The key empirical demonstration is that residual networks achieve lower training error than plain networks under the exact same optimization procedure. The training hyperparameters are not tuned separately for plain versus residual networks β€” they are held constant. This is essential: if residual networks required special learning rates or schedules to work, the claim that they are "easier to optimize" would be weaker. The fact that they succeed under standard settings that fail for plain networks is precisely the evidence.

Design Rationale: Why Identity Shortcuts Over Alternatives

The paper's most important architectural decision is the choice of identity mapping over learned transformations for the shortcut connections. This is not an arbitrary choice β€” it follows directly from the degradation problem analysis and has specific optimization consequences.

Why not gated shortcuts (highway networks)? Highway networks (Srivastava et al., 2015) use learned gating functions that determine, for each input element, what fraction passes through the shortcut (the "carry gate") and what fraction is transformed (the "transform gate"). These gates are data-dependent sigmoid functions with their own parameters. The paper identifies two key disadvantages:

  1. The gate can close: When the transform gate is saturated toward 1 and the carry gate toward 0, the layers in the highway block behave exactly like a plain network β€” they must learn the full function from scratch. The network can thus opt out of residual learning in some or all blocks, which defeats the preconditioning benefit. The empirical evidence (highway networks not demonstrating gains beyond ~100 layers) suggests that learned gating does not provide the same optimization benefits as hardwired identity.

  2. Extra parameters: Gated shortcuts add parameters for the gating mechanism, which makes it harder to isolate the effect of residual learning from the effect of additional capacity. The ResNet's parameter-free identity shortcuts enable clean comparisons with plain networks.

Why not always use projection shortcuts (option C)? Table 3 shows that using projection shortcuts everywhere (option C) is marginally better than using them only for dimension matching (option B), and both are only marginally better than using zero-padding shortcuts everywhere (option A). The key insight is that the large gap is between any residual architecture and the plain baseline β€” the differences among residual variants are small. This has two implications:

  1. The residual formulation, not projection parameters, drives the improvement. Zero-padding shortcuts are parameter-free, yet they already solve the degradation problem. The additional gains from projection shortcuts are minor and attributable simply to having more parameters.

  2. Identity shortcuts are sufficient and economical. Using identity shortcuts wherever possible reduces model size and computational cost, which is important for practical deployment and for fair comparisons.

Why bias toward identity rather than zero? An alternative formulation could define the residual as $\mathcal{F}(x) = \mathcal{H}(x)$ (learn the full function) with an additive identity shortcut, making the output $\mathcal{F}(x) + x$. This is mathematically equivalent to the paper's formulation β€” the actual function computed is the same β€” but the optimization dynamics differ. In the paper's formulation $\mathcal{F}(x) = \mathcal{H}(x) - x$, the target for $\mathcal{F}$ is zero when identity is optimal, which biases weights toward zero (a natural basin of attraction under weight decay). If instead $\mathcal{F}(x)$ were the full function and identity came from the shortcut, the target for $\mathcal{F}$ when identity is optimal would be zero as well (since $\mathcal{H}(x) = x$ would require $\mathcal{F}(x) = 0$), so the formulations converge. The key difference is not in the mathematical equivalence but in the conceptual framing: the paper explicitly formulates residual learning as learning the deviation from identity, which motivates the empirical analysis of layer responses (Figure 7, discussed below) showing that residual functions indeed have smaller magnitudes than plain network activations.

Why the identity shortcut must be connected before the final ReLU. The paper specifies that after element-wise addition $\mathcal{F}(x) + x$, a ReLU is applied: $\sigma(\mathcal{F}(x) + x)$. If the ReLU were applied to $\mathcal{F}(x)$ before addition (i.e., $y = \sigma(\mathcal{F}(x)) + x$), the residual signal would be non-negative, which would restrict the representational capacity β€” the output could only be greater than or equal to the input in each dimension. By applying ReLU after addition, the residual $\mathcal{F}(x)$ can take any real value (positive or negative), and the nonlinearity is applied to the combined signal.

Analysis of Layer Responses: Empirical Validation of the Preconditioning Hypothesis

After training, the paper analyzes the magnitudes of layer responses to test whether residual learning actually produces the behavior predicted by the preconditioning hypothesis β€” namely, that residual functions are closer to zero (i.e., represent smaller perturbations from identity) than the corresponding plain network functions.

What is measured (Figure 7). The standard deviation (std) of the output of each 3Γ—3 convolutional layer is computed, taken after batch normalization and before the nonlinearity (ReLU for plain networks, ReLU or addition for residual networks). For ResNets, this measures the response strength of the residual function $\mathcal{F}(x)$ before it is added to the shortcut β€” it quantifies how much the block modifies its input. The std is plotted in two ways: sorted by magnitude (top panel) and in original layer order from input to output (bottom panel).

Key observations from Figure 7:

  1. ResNets have generally smaller responses than plain counterparts. Across all network depths (20, 56, 110 layers), the response standard deviations for ResNets are consistently lower than for the corresponding plain networks. This directly supports the hypothesis that residual functions learn smaller perturbations β€” they modify the signal less dramatically than plain layers, consistent with the idea that identity mappings provide a reasonable starting point and only small corrections are needed.

  2. Deeper ResNets have smaller responses. Comparing ResNet-20, ResNet-56, and ResNet-110, the std values decrease as depth increases. The paper interprets this as: "When there are more layers, an individual layer of ResNets tends to modify the signal less." In a very deep residual network, each block makes only a tiny adjustment to the representation, with the cumulative effect of many small adjustments producing the final output.

  3. Plain networks show no such pattern. Plain-20 and plain-56 have similar response magnitudes (and plain-56 has higher training error, indicating that the larger responses are not necessarily productive).

Why this analysis matters: It provides mechanistic evidence for the paper's central hypothesis. If residual learning truly works by preconditioning the optimization problem such that the target function is close to identity, then trained residual functions should be "small" in some sense β€” they should represent modest adjustments to the input rather than wholesale transformations. Figure 7 confirms this prediction, distinguishing the ResNet paper's contribution from a mere architectural trick. The finding that deeper ResNets exhibit smaller per-layer responses is particularly striking β€” it suggests that depth in residual networks works by decomposing the overall transformation into many tiny steps, each of which individually modifies the signal only slightly, making optimization manageable even when the total transformation is complex.

4. Key Insights and Innovations

Innovation 1: Reframing Depth as an Optimization Problem, Not a Capacity Problem

The paper's most fundamental conceptual move is redefining why deep networks fail. Before ResNet, the dominant narrative was that deeper networks might overfit or that vanishing/exploding gradients prevented convergence. Both explanations pointed toward insufficient capacity or broken signal propagation β€” problems that better initialization and normalization were already solving. The ResNet paper makes a sharp diagnostic pivot: the degradation problem (Figure 1) persists even when gradients flow healthily, meaning the solver cannot find good solutions that provably exist in the hypothesis space.

This is not merely a new observation β€” it is a reframing of the entire challenge of deep learning. The paper argues, through a constructive proof, that if a shallower network trains to some error, a deeper version of that same network can be initialized to achieve exactly that error (by setting new layers to identity and copying shallow weights). Therefore, any training error increase with depth is a solver failure, not a representational shortcoming. Prior work had treated depth difficulties as vaguely architectural (VGG just stopped at 19 layers, GoogLeNet at 22) without this crisp diagnostic. Even highway networks (Srivastava et al., 2015), the closest concurrent work, framed their contribution as enabling information flow through depth rather than diagnosing why standard SGD fails on identity-like solutions.

The significance of this reframing is that it changes what "solving depth" means. Before ResNet, the implicit goal was designing architectures where gradients don't vanish or where capacity is well-managed. After ResNet, the goal becomes: how do we precondition the optimization landscape so that SGD finds solutions that are at least as good as shallow baselines? This shifts the design space from "build a better propagation mechanism" to "build a better loss surface." The residual formulation is not just a new architecture among many β€” it is the consequence of this diagnostic insight, and that is what makes it intellectually distinctive.

Evidence for this reframing's validity comes from the training curves themselves (Figures 4 and 6, left panels): plain-34 consistently has higher training error than plain-18 throughout the entire training trajectory, despite having strictly more capacity. The paper explicitly rules out vanishing gradients with BN normalization checks. The empirical fact that a larger hypothesis space produces a worse solution under identical optimization is the core puzzle that residual learning resolves.

Innovation 2: Hardwiring Identity as a Structural Inductive Bias Rather Than a Learned Behavior

Where highway networks allow identity-like behavior through learned gating mechanisms (the transform gate can saturate toward carry), ResNets mandate it through parameter-free identity shortcuts. This is not a minor implementation detail β€” it represents a fundamentally different philosophy about how to embed prior knowledge into neural architectures.

The key intellectual distinction: highway networks treat the shortcut as optional β€” the network learns whether to use it. ResNets treat the shortcut as structural β€” the network cannot opt out. Every block computes F(x) + x, with the identity path always operating at full strength. The residual function F(x) learns only the deviation from identity, never the full transformation. This is the architectural equivalent of changing the prior: instead of starting with a blank slate and hoping the optimizer discovers that identity-ish solutions work well, ResNets bake identity into the computation graph and force the optimizer to work relative to it.

Why is this a conceptual advance rather than just a better architectural choice? Because it demonstrates that strong, non-learnable inductive biases can be more effective than flexible, learnable ones β€” a finding that runs counter to the "let the network learn everything" ethos of end-to-end deep learning. The highway network designers presumably believed that data-dependent gating would be more powerful because the network could adaptively choose when to use shortcuts. The ResNet paper shows the opposite: the rigidity of identity shortcuts is precisely what makes them work, because it constrains the optimization trajectory to stay near the identity manifold, preventing the solver from wandering into ill-conditioned regions of parameter space.

The empirical proof of this distinction is direct but understated: highway networks "have not demonstrated accuracy gains with extremely increased depth (e.g., over 100 layers)" (Section 2). ResNets successfully train at 152 layers on ImageNet and 1202 layers on CIFAR-10. The difference between "can train 100+ layers" and "can train 30 layers" is not quantitative β€” it is qualitative. It means hardwired identity unlocks a regime of depth that learned-gating approaches cannot reach, suggesting the inductive bias is doing something more fundamental than just providing an alternative gradient path.

Innovation 3: The Degradation Problem as a First-Class Empirical Phenomenon

Prior to this paper, the fact that deeper networks sometimes underperformed shallower ones was noted in passing (He and Sun, 2015; Srivastava et al., 2015) but was never systematically characterized or given a name. The ResNet paper elevates degradation to a named, diagnosed, and experimentally isolated phenomenon β€” the "degradation problem" β€” and makes it the central antagonist that residual learning defeats.

What makes this an innovation, not just a label? Three things:

First, the paper provides the cleanest experimental isolation of degradation from confounds. By comparing plain networks that differ only in depth (same architecture family, same training procedure, same initialization), the paper rules out competing explanations: overfitting (training error degrades, not just test error), vanishing gradients (BN ensures healthy gradient norms), insufficient training time (3Γ— iterations don't close the gap), or poor initialization (MSRA initialization is already well-tuned). What remains is a pure signal: deeper networks are harder to optimize, period. Figures 1, 4, and 6 (left panels) are the canonical visualizations of this phenomenon β€” the deeper training error curve starting higher and staying higher than the shallower one, with no crossing point.

Second, the paper connects degradation to a conceptual argument about solution existence. The constructed-solution thought experiment β€” "if you can train a shallow network, you can construct a deep network that does no worse by setting new layers to identity" β€” transforms degradation from a vague "deeper is harder" into a precise "SGD cannot find identity mappings in deep nonlinear stacks." This is a testable hypothesis, not just a lament, and it directly motivates the residual reformulation.

Third, the degradation problem becomes the diagnostic tool for evaluating solutions. Throughout the paper, the key metric is not just final accuracy but whether training error decreases with depth. The residual networks' success is demonstrated primarily through the reversal of degradation: ResNet-34 has lower training error than ResNet-18 (Figure 4, right), whereas plain-34 has higher training error than plain-18 (Figure 4, left). The degradation problem serves as the conceptual yardstick β€” any architecture that claims to enable depth must first demonstrate that training error monotonically improves with more layers.

This contribution may seem obvious in retrospect, but naming a phenomenon is a genuine intellectual act. Before ResNet, the community had a fuzzy sense that depth was tricky; after ResNet, we had a precise, named phenomenon with a known cause and a tested solution. That diagnostic clarity is what enabled the explosion of deep architecture research in subsequent years.

Innovation 4: Empirical Discovery That Residual Functions Are Genuinely Small

Section 3.1 hypothesizes that residual learning works because identity provides good preconditioning β€” the optimal function is "closer to an identity mapping than to a zero mapping," so learning perturbations from identity is easier than learning the full function. This hypothesis is intuitive but could be wrong β€” residual networks might work for unrelated reasons (better gradient flow, ensembling effects from multiple paths, etc.). The paper provides direct mechanistic evidence for the preconditioning hypothesis through the layer response analysis in Figure 7, showing that residual functions have genuinely smaller magnitudes than plain network activations.

This is an innovation in how to validate an architectural hypothesis. Rather than resting on "it works, therefore the hypothesis is right," the paper opens up the trained networks and measures their internal behavior. The finding that ResNet layer responses have lower standard deviation than plain counterparts β€” and that deeper ResNets have progressively smaller per-layer responses β€” is not a necessary consequence of the residual formulation. One could imagine residual functions that learn large perturbations from identity, making the identity shortcut essentially irrelevant. The fact that trained ResNets actually exhibit small residuals (the modification to the signal at each block is modest) confirms that the hypothesized mechanism is real, not just a post-hoc story.

The significance extends beyond this paper. This analysis technique β€” measuring internal activation statistics to validate optimization hypotheses β€” became a template for future architectural research. It demonstrates that one can move beyond "architecture A achieves X% accuracy" to "architecture A exhibits this measurable internal property, which we hypothesized would improve optimization, and that property indeed correlates with better training." The Figure 7 analysis transforms the ResNet claim from an empirical observation ("residual networks train deeper") into a mechanistic understanding ("residual networks train deeper because each layer makes only a small correction to a mostly-correct identity pathway, keeping the optimization problem well-conditioned at scale").

This finding also resolves a potential paradox. The residual formulation is mathematically equivalent to a plain network with clever weight initialization (set weights near zero so the block approximates identity). Why not just initialize plain networks to near-identity? The layer response analysis suggests why: in a plain network, there is no structural pressure to stay near identity during training. The weights might start near zero but drift into regions where the block computes large transformations, entering ill-conditioned optimization regimes. The identity shortcut in ResNets provides a persistent structural bias: even as weights adapt, the network always has the option to fall back toward identity for each block, keeping the overall function anchored. The small response magnitudes in Figure 7 are evidence that this anchoring actually occurs in practice.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary dataset for classification experiments is the ImageNet 2012 classification dataset (Russakovsky et al., 2015), consisting of 1000 object classes, 1.28 million training images, 50k validation images, and 100k test images (evaluated via the official test server for final results). A secondary dataset, CIFAR-10 (Krizhevsky, 2009), consists of 50k training images and 10k testing images across 10 classes, used for controlled analysis of extremely deep network behavior. For object detection, the paper uses PASCAL VOC 2007 and 2012 (Everingham et al., 2010) and MS COCO (Lin et al., 2014).

  • Base model(s). The paper constructs custom residual network architectures (ResNets) in five depth variants: 18, 34, 50, 101, and 152 layers for ImageNet, plus 20, 32, 44, 56, 110, and 1202 layers for CIFAR-10. These are compared against plain network counterparts of identical depth, width, and parameter count β€” the same architectures minus the shortcut connections. For external comparison, the paper benchmarks against VGG-16 and VGG-19 (Simonyan and Zisserman, 2014), GoogLeNet (Szegedy et al., 2015), and PReLU-net (He et al., 2015). For detection baselines, Faster R-CNN (Ren et al., 2015) with either VGG-16 or ResNet-101 serves as the detection framework.

  • Metrics. ImageNet classification is evaluated using top-1 and top-5 error rates (percentage of images where the correct class is not the top predicted class, or not among the top 5, respectively). CIFAR-10 uses classification error (percentage misclassified). Object detection on PASCAL VOC uses mean Average Precision at IoU = 0.5 (mAP@.5); on COCO, both the PASCAL-style mAP@.5 and the standard COCO metric mAP@[.5, .95] (averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05) are reported.

  • Baselines. For ImageNet classification: plain-18 and plain-34 (identical architectures without shortcuts), VGG-16 (as reimplemented/tested by the authors, 28.07% top-1 error), GoogLeNet (9.15% top-5 error as reported), and PReLU-net (24.27% top-1 error). For CIFAR-10: plain-20, plain-32, plain-44, plain-56 counterparts to the ResNets, plus external baselines from the literature including Maxout (Goodfellow et al., 2013), NIN (Lin et al., 2013), DSN (Lee et al., 2014), FitNet (Romero et al., 2015), and Highway networks (Srivastava et al., 2015a,b). For detection: Faster R-CNN with VGG-16 serves as the primary baseline.

  • Computational budget / compute accounting. Computational cost is measured in FLOPs (floating-point operations, specifically multiply-add operations), reported for each architecture in Table 1. For example, the 34-layer plain and residual networks each require 3.6 billion FLOPs; ResNet-152 requires 11.3 billion FLOPs (lower than VGG-19 at 19.6 billion). The paper emphasizes that identity shortcuts add neither extra parameters nor computational complexity beyond the negligible element-wise addition, enabling direct, fair comparisons between plain and residual networks of identical parameter count and computational budget.

  • Cross-validation / statistical protocol. For CIFAR-10, the learning rate schedule was determined on a 45k/5k train/val split of the 50k training data, after which the full training set was used for final models. For ResNet-110 on CIFAR-10, the paper runs the experiment 5 times and reports both best and mean Β± standard deviation, following the practice of the Highway networks paper. For ImageNet validation results, single-model numbers are reported without confidence intervals; the test server results (Table 5) are single-point evaluations. The paper does not employ k-fold cross-validation or statistical significance testing on ImageNet classification results.

Main Quantitative Results

The results are organized into three experiment families: ImageNet classification (the primary demonstration), CIFAR-10 analysis (controlled depth scaling and mechanistic studies), and object detection on PASCAL VOC and MS COCO (transfer learning validation).

The Degradation Problem: Characterized and Diagnosed

The paper first establishes the degradation problem empirically before showing that residual learning solves it.

ImageNet plain networks (Figure 4, left; Table 2). The 34-layer plain network achieves 28.54% top-1 error on ImageNet validation, compared to 27.94% for the 18-layer plain network under 10-crop testing β€” a 0.60 percentage point increase in error despite having greater capacity. Figure 4 (left) shows the training dynamics: the 34-layer plain network exhibits higher training error than the 18-layer version throughout the entire training procedure. The thin curves (training error) for plain-34 remain above those for plain-18 from start to finish, and the bold curves (validation error) mirror this gap. This is critical because it rules out overfitting β€” the deeper network is worse on the very data it is trained on.

CIFAR-10 plain networks (Figure 6, left). The pattern replicates at smaller scale with controlled depth variation. As depth increases from 20 to 32 to 44 to 56 layers, the training error increases monotonically. The dashed training error curves in Figure 6 (left) show plain-20 achieving the lowest training error, plain-56 the highest, with plain-32 and plain-44 ordered between them. The deep plain nets "suffer from increased depth, and exhibit higher training error when going deeper." The 110-layer plain network is so difficult to optimize that its error exceeds 60% and is not displayed in the figure. The paper notes that "such an optimization difficulty is a fundamental problem" β€” it appears on both ImageNet and CIFAR-10, at different scales, with different network widths, confirming it is not dataset-specific or an artifact of a particular architecture scale.

Additional training iterations don't help. In a footnote, the paper notes having "experimented with more training iterations (3Γ—) and still observed the degradation problem, suggesting that this problem cannot be feasibly addressed by simply using more iterations." This is important because it distinguishes degradation from simply slower convergence β€” the deeper network would need exponentially more compute to match the shallower one, which is infeasible.

Gradients are healthy. The paper explicitly checks and verifies that with batch normalization, "neither forward nor backward signals vanish." The 34-layer plain network "is still able to achieve competitive accuracy" (Table 3 shows plain-34 achieving 28.54% top-1 error, competitive with earlier methods). This confirms that degradation is not a vanishing-gradient problem and represents a qualitatively different optimization difficulty β€” the authors "conjecture that the deep plain nets may have exponentially low convergence rates."

Residual Networks Reverse Degradation (Tables 2–3, Figures 4 and 6)

The paper's central empirical claim is that adding identity shortcut connections to create residual networks makes depth beneficial rather than detrimental.

ImageNet ResNets (Figure 4, right; Table 2). The 34-layer ResNet achieves 25.03% top-1 error, compared to 27.88% for the 18-layer ResNet β€” a 2.85 percentage point improvement from increased depth. This is the opposite of the plain network result (where deeper was worse). Figure 4 (right) shows the training dynamics: the 34-layer ResNet has lower training error than the 18-layer ResNet throughout training, and this advantage transfers to validation error. The paper states: "the 34-layer ResNet exhibits considerably lower training error and is generalizable to the validation data. This indicates that the degradation problem is well addressed in this setting and we manage to obtain accuracy gains from increased depth."

Comparing plain vs. residual directly (Table 2). The 34-layer results are particularly striking: plain-34 achieves 28.54% while ResNet-34 achieves 25.03% β€” a 3.51 percentage point reduction in top-1 error. Since both networks have exactly the same parameters, depth, width, and FLOPs (the identity shortcuts add zero parameters), this gap is attributable solely to the residual formulation enabling better optimization. For the 18-layer networks, plain and residual are comparable (27.94% vs. 27.88%), but Figure 4 (right vs. left) shows that even at this moderate depth, "the 18-layer ResNet converges faster" β€” the training error curve drops more quickly in the early iterations.

CIFAR-10 ResNets (Figure 6, middle; Table 6). The controlled depth sweep confirms the pattern. ResNet-20: 8.75% error. ResNet-32: 7.51%. ResNet-44: 7.17%. ResNet-56: 6.97%. ResNet-110: 6.43% (best, with mean Β± std of 6.61% Β± 0.16 over 5 runs). Error decreases monotonically with depth β€” the opposite of the plain network trend. Figure 6 (middle) shows the training curves: the deeper ResNets achieve strictly lower training error than the shallower ones, and the validation error follows suit. "Our ResNets manage to overcome the optimization difficulty and demonstrate accuracy gains when the depth increases."

The 110-layer network requires a learning rate warmup. The initial learning rate of 0.1 is "slightly too large to start converging" for the 110-layer ResNet, so the authors use 0.01 for a brief warmup period (~400 iterations, until training error drops below 80%) before switching to 0.1. The paper notes this is a minor adjustment β€” even without warmup, the network still converges to similar accuracy but takes longer to stabilize initially.

Identity vs. Projection Shortcuts (Table 3)

To determine whether the residual mechanism or the additional projection parameters drive the improvement, the paper ablates three shortcut options on the 34-layer architecture:

  • Option A (zero-padding, zero extra parameters): 25.03% top-1 error, 7.76% top-5 error.
  • Option B (projection only for dimension increases): 24.52% top-1, 7.46% top-5.
  • Option C (all shortcuts are projections): 24.19% top-1, 7.40% top-5.

All three options are "considerably better than the plain counterpart" (plain-34: 28.54% top-1). The gap between the best plain network (18-layer: 27.94%) and the worst residual variant (option A: 25.03%) is approximately 2.9 percentage points β€” a substantial margin. The improvements from B over A (0.51 points) and C over B (0.33 points) are modest. The paper draws the key conclusion: "projection shortcuts are not essential for addressing the degradation problem." Identity shortcuts with zero-padding (option A) already provide the primary benefit, confirming that the residual formulation itself β€” not additional learned transformations β€” is what enables deep training.

The paper uses option B for ResNet-50/101/152 as a minor accuracy improvement, and option A for CIFAR-10 experiments (to maintain exact parameter parity with plain networks).

Deeper Bottleneck Architectures: Scaling to 50, 101, and 152 Layers (Tables 3–5)

Replacing the basic two-layer blocks in the 34-layer architecture with the three-layer bottleneck design (1Γ—1, 3Γ—3, 1Γ—1 convolutions) enables substantially deeper networks while maintaining computational tractability.

ResNet-50 (Table 1, 3.8 billion FLOPs): 22.85% top-1 error, a 1.67 percentage point improvement over ResNet-34 B (24.52%). The 50-layer variant gains accuracy from the additional depth and bottleneck design.

ResNet-101 (Table 1, 7.6 billion FLOPs): 21.75% top-1 error. Compared to ResNet-50, adding 51 more layers (primarily in conv4_x, which expands from 6 to 23 bottleneck blocks) yields a 1.10 percentage point improvement. There is no sign of degradation β€” deeper continues to help.

ResNet-152 (Table 1, 11.3 billion FLOPs): 21.43% top-1 error. The deepest variant, at 8Γ— the depth of VGG-19, reduces error by another 0.32 points over ResNet-101. Notably, the 152-layer ResNet (11.3 billion FLOPs) still has lower computational complexity than VGG-16 (15.3 billion) or VGG-19 (19.6 billion). The paper emphasizes: "We do not observe the degradation problem and thus enjoy significant accuracy gains from considerably increased depth. The benefits of depth are witnessed for all evaluation metrics."

Single-model results compared to prior state of the art (Table 4). Under the more comprehensive multi-scale fully-convolutional testing protocol, the single-model results are:

  • ResNet-34 B: 21.84% top-1, 5.71% top-5 (already competitive with BN-inception at 21.99%/5.81% and PReLU-net at 21.59%/5.71%)
  • ResNet-50: 20.74% top-1, 5.25% top-5
  • ResNet-101: 19.87% top-1, 4.60% top-5
  • ResNet-152: 19.38% top-1, 4.49% top-5

The 152-layer single model's 4.49% top-5 error "outperforms all previous ensemble results" (Table 5), including VGG's ensemble at 6.8% and GoogLeNet's ensemble at 7.89% (ILSVRC 2014 results on the test set).

Ensemble result (Table 5). An ensemble of six models of varying depth (only two of which were 152-layer at submission time) achieves 3.57% top-5 error on the ImageNet test set, winning the 1st place in ILSVRC 2015 classification. This represents a substantial reduction from the previous year's winner (GoogLeNet at 6.66% test error).

CIFAR-10 Analysis: Pushing Past 1000 Layers (Table 6, Figure 6)

Beyond the standard depth sweeps, the paper explores an "aggressively deep model" of 1202 layers (n = 200 in the CIFAR-10 architecture formula, yielding 6Γ—200+2 = 1202 weighted layers with 19.4M parameters).

Training behavior (Figure 6, right). The 1202-layer ResNet achieves training error <0.1%, demonstrating "no optimization difficulty." The training curve converges smoothly and reaches near-zero error, confirming that residual learning scales to extreme depths without degradation.

Test performance (Table 6). The 1202-layer ResNet achieves 7.93% test error β€” worse than the 110-layer ResNet's 6.43%, despite having substantially lower training error. The paper diagnoses this as overfitting: "The 1202-layer network may be unnecessarily large (19.4M) for this small dataset." The 110-layer ResNet (1.7M parameters) provides a better capacity match for CIFAR-10's 50k training images. The paper explicitly notes that no maxout or dropout was used β€” regularization was solely through "deep and thin architectures by design" β€” and that combining with stronger regularization may improve the 1202-layer result.

Comparison with prior CIFAR-10 methods (Table 6). The 110-layer ResNet at 6.43% is among state-of-the-art results. This is achieved with 1.7M parameters β€” less than FitNet (2.5M) and comparable to Highway networks (1.25M for 32-layer, 2.3M for 19-layer). The 32-layer ResNet (0.46M parameters, 7.51% error) slightly outperforms the 19-layer Highway network (2.3M parameters, 7.54%).

Layer Response Analysis (Figure 7)

The paper measures the standard deviation (std) of layer outputs (after BN, before nonlinearity) across all 3Γ—3 convolutional layers in trained CIFAR-10 networks.

ResNets vs. plain networks (Figure 7, top and bottom). In both the sorted view (responses ordered by magnitude) and the original-order view (layers from input to output), ResNet responses have smaller standard deviations than their plain counterparts at matched depths. For example, ResNet-20 shows lower std values than plain-20, and ResNet-56 shows lower std than plain-56, across nearly all layers.

Depth trend within ResNets (Figure 7). Among residual networks, the response magnitude decreases with depth: ResNet-20 has the largest std values, ResNet-56 intermediate, and ResNet-110 the smallest. The paper interprets: "When there are more layers, an individual layer of ResNets tends to modify the signal less." This supports the preconditioning hypothesis β€” in a very deep residual network, each block makes only a tiny adjustment, with the cumulative effect of many small adjustments producing the final output. The monotonically decreasing per-layer response with depth also implies that the optimization burden per residual block becomes progressively lighter, explaining why extreme depths remain trainable.

Plain networks show no depth-related trend. Plain-20 and plain-56 have similar response magnitude distributions, despite plain-56 suffering from degradation. The plain network's layers don't naturally learn smaller transformations as depth increases, consistent with the hypothesis that plain networks lack the structural bias toward identity that would encourage modest per-layer modifications.

Object Detection on PASCAL VOC and MS COCO (Tables 7–11)

The paper tests whether the improved representations from residual networks transfer to object detection, using Faster R-CNN as the detection framework with either VGG-16 or ResNet-101 as the backbone.

PASCAL VOC detection (Table 7). ResNet-101 improves detection mAP@.5 over VGG-16 by:

  • VOC 2007 test set (trained on VOC 07+12): 76.4% vs. 73.2% (+3.2 percentage points)
  • VOC 2012 test set (trained on VOC 07++12): 73.8% vs. 70.4% (+3.4 percentage points)

COCO detection (Table 8). On the COCO validation set, ResNet-101 improves:

  • mAP@.5: 48.4% vs. 41.5% (+6.9 percentage points)
  • mAP@[.5, .95]: 27.2% vs. 21.2% (+6.0 percentage points, a 28% relative improvement)

The paper highlights: "Remarkably, the mAP@[.5, .95]'s absolute increase (6.0%) is nearly as big as mAP@.5's (6.9%). This suggests that a deeper network can improve both recognition and localization." The COCO metric is particularly demanding because it averages over multiple IoU thresholds, so improvement here indicates that bounding boxes are both more accurately classified and better localized.

Detection implementation is identical between backbones. The paper carefully notes: "The detection implementation (see appendix) of using both models is the same, so the gains can only be attributed to better networks." This isolates the representation quality as the sole source of improvement.

Competition results (Tables 9–12). With additional improvements (box refinement, global context, multi-scale testing, ensemble), ResNet-101-based detection systems won 1st place in:

  • COCO detection (Table 9): 59.0% mAP@.5, 37.4% mAP@[.5, .95] on test-dev (ensemble)
  • PASCAL VOC 2012 (Table 11): 83.8% mAP (single model with COCO pretraining and improvements), 10 points higher than the previous state of the art
  • ImageNet detection (Table 12): 62.1% mAP on the DET test set (ensemble), surpassing the second place by 8.5 points absolute

ImageNet localization (Tables 13–14). Using a per-class region proposal network (RPN) with ResNet-101:

  • Oracle testing (ground truth class given): 11.7% localization error with dense multi-scale testing, compared to VGG-16's 33.1% center-crop error β€” a ~65% relative reduction
  • With predicted classes: 14.4% top-5 localization error using ResNet-101 for classification
  • With R-CNN refinement: 10.6% single-model validation error
  • Ensemble: 9.0% top-5 localization error on the test set, a 64% relative reduction over the ILSVRC 2014 winning result (VGG at 25.3%)

Ablation Studies and Robustness Checks

  • Degradation across datasets (Figures 1, 4, 6): The degradation problem β€” deeper plain networks having higher training error than shallower ones β€” is replicated on CIFAR-10 (Figure 1 and Figure 6 left), ImageNet (Figure 4 left), and the paper cites MNIST results from the Highway networks paper, establishing that the phenomenon is not dataset-specific.

  • Degradation is not solved by more training (Section 4.1 footnote): Three times the standard training iterations on ImageNet plain-34 still exhibit degradation β€” the deeper network never catches up to the shallower one, ruling out slow convergence as the sole explanation.

  • Zero-padding vs. projection shortcuts (Table 3): All three shortcut options (A: zero-padding, B: projection for dimension changes, C: all projections) substantially outperform the plain baseline, with B slightly better than A (+0.51 top-1 points) and C marginally better than B (+0.33 points). The large gap is between any residual variant and the plain network, not between residual variants, confirming that identity shortcuts with zero-padding are sufficient to solve degradation β€” projection parameters provide only incremental gains.

  • Bottleneck design efficiency (Table 1, FLOPs column): The bottleneck block enables 152-layer training at 11.3 billion FLOPs β€” less computation than shallower VGG nets. The paper notes that deeper non-bottleneck ResNets would also gain accuracy (as shown on CIFAR-10) but are "not as economical," so the bottleneck choice is an engineering optimization, not a fundamental requirement for residual learning to work.

  • Single-layer residual block is ineffective (Section 3.2): The paper reports that a residual block with only a single layer (y = W₁x + x) provides no observed advantages, suggesting that at least two nonlinear layers per block are needed for the residual formulation to be beneficial.

  • CIFAR-10 parameter parity (Table 6, Section 4.2): All CIFAR-10 residual and plain architectures use identity shortcuts with zero-padding (option A) so that "our residual models have exactly the same depth, width, and number of parameters as the plain counterparts." This eliminates parameter count as a confound β€” any accuracy differences must come from optimization ease, not model capacity.

  • ResNet-110 learning rate warmup (Section 4.2): For the 110-layer ResNet on CIFAR-10, a brief warmup at 0.01 learning rate (until training error <80%, ~400 iterations) is used before switching to the standard 0.1 schedule. The paper notes that without warmup, the network "starts converging (<90% error) after several epochs, but still reaches similar accuracy" β€” the warmup is a stability aid, not a performance-critical requirement.

  • Overfitting at 1202 layers (Figure 6 right, Table 6): The 1202-layer ResNet achieves <0.1% training error but 7.93% test error, worse than the 110-layer's 6.43%. This is a clear overfitting signal, attributed to the model being "unnecessarily large" for CIFAR-10. The paper did not use dropout or maxout, relying solely on architectural depth and narrowness for regularization β€” a design choice to avoid distracting from the optimization focus.

  • Detection improvement is solely from representations (Tables 7–8): The detection results are obtained by simply swapping the backbone (VGG-16 β†’ ResNet-101) within an identical Faster R-CNN framework with the same hyperparameters. The 28% relative COCO improvement is therefore attributable solely to the quality of the features learned by the residual network, not to detection-specific innovations.

Critical Assessment

Claim: Residual networks solve the degradation problem (deeper networks achieve lower training error). This claim is strongly and directly supported by the training curves in Figures 4 (right) and 6 (middle). On both ImageNet and CIFAR-10, the deeper ResNet variants have lower training error than their shallower counterparts throughout the training procedure, contrasting with the plain network behavior where deeper is worse (Figures 4 left, 6 left). The evidence is unusually clean because the comparison controls for parameter count, width, depth, and training procedure β€” the only difference is the presence of identity shortcuts. The claim is demonstrated at multiple depths (18 vs. 34 on ImageNet; 20 vs. 32 vs. 44 vs. 56 vs. 110 on CIFAR-10), across two datasets with different scales, and with both basic-block and bottleneck-block architectures (Tables 3–4). There is no evidence of degradation in any residual configuration tested.

However, a nuance: the degradation problem is only demonstrated for plain networks up to 56 layers (CIFAR-10) and 34 layers (ImageNet). The paper does not show how a plain-50, plain-101, or plain-152 bottleneck network would behave. The degradation claim for bottleneck architectures is stated in a footnote β€” "the degradation problem of plain nets is also witnessed for the bottleneck designs" β€” but no data is shown. This is a minor gap: the reader must trust that bottleneck plain networks also degrade, without seeing the training curves.

Claim: Identity shortcuts (not learned projections) are what matter. This claim is well-supported by the ablation in Table 3. Option A (zero-padding, zero extra parameters) achieves 25.03% top-1 error vs. plain-34's 28.54% β€” a 3.51 point gap with no additional parameters. Options B and C add projection parameters but provide only 0.51 and 0.33 additional points of improvement, respectively. The paper correctly concludes that projections are "not essential for addressing the degradation problem." However, the ablation is only shown for ResNet-34 on ImageNet. It would strengthen the argument to see the same ablation at other depths (e.g., does zero-padding still suffice for ResNet-101?) or on CIFAR-10, but the paper reasonably assumes the conclusion generalizes given that CIFAR-10 experiments (option A only) successfully train 110- and 1202-layer networks.

Claim: Residual learning enables accuracy gains from substantially increased depth. This claim is strongly supported for the tested depth range. On ImageNet, top-1 error monotonically decreases: 18-layer (27.88%) β†’ 34-layer (25.03%) β†’ 50-layer (22.85%) β†’ 101-layer (21.75%) β†’ 152-layer (21.43%) under 10-crop testing (Tables 2–3), and similarly under multi-scale testing (Table 4). On CIFAR-10: 20-layer (8.75%) β†’ 32-layer (7.51%) β†’ 44-layer (7.17%) β†’ 56-layer (6.97%) β†’ 110-layer (6.43%). The improvements are consistent in direction, though diminishing β€” the marginal benefit of each additional layer decreases, which is expected.

A genuine limitation: there is no experiment showing when depth gains saturate or degrade for residual networks. The 1202-layer CIFAR-10 result (7.93%, worse than 110-layer's 6.43%) is attributed to overfitting, not optimization difficulty, so it doesn't answer the question of whether a residual network on a sufficiently large dataset would eventually suffer from degradation at some depth (2000 layers? 5000 layers?). The paper establishes that the degradation problem is pushed well beyond practical depths but doesn't characterize the new limit.

Claim: Residual functions are genuinely smaller (closer to zero), supporting the preconditioning hypothesis. This claim is supported by Figure 7 with clear visual evidence. ResNet layer responses have lower standard deviation than plain counterparts, and deeper ResNets have progressively smaller per-layer responses. However, the analysis has limitations: (a) it is shown only on CIFAR-10 (not ImageNet), (b) it uses standard deviation as a proxy for "closeness to zero" without showing the actual distribution of response values (are the responses symmetrically distributed around zero with small variance, or are they small but biased?), and (c) it is correlational β€” it shows that small residuals occur in well-optimized networks but doesn't prove that small residuals cause the optimization benefit. The hypothesis that identity preconditioning helps optimization because residuals are naturally small is plausible and consistent with the evidence, but the evidence is not a direct causal test.

Claim: The learned representations generalize well to other recognition tasks. This claim is strongly supported by the detection and localization results. The COCO mAP@[.5, .95] improvement of 28% relative over VGG-16 (Table 8) using the exact same detection framework is compelling because it isolates representation quality as the causal factor. The sweep of competition wins across five tracks (ImageNet classification, detection, localization; COCO detection, segmentation) suggests the representations are broadly useful, though the paper doesn't test generalization to domains substantially different from object recognition (e.g., video, medical imaging, language).

Potential weaknesses and missing experiments:

  • Single training run per configuration on ImageNet. The paper reports point estimates for ImageNet error rates without confidence intervals, standard deviations, or multiple runs. Given the stochasticity in SGD training, it is unclear whether the reported differences (e.g., ResNet-152 at 21.43% vs. ResNet-101 at 21.75%) are statistically reliable or within noise. CIFAR-10 does provide error bars for ResNet-110 (5 runs), and these show a standard deviation of Β±0.16 points β€” small relative to the depth-induced gains but potentially large enough to affect the ordering of close comparisons.

  • No direct comparison with highway networks on ImageNet. The paper critiques highway networks for not demonstrating gains beyond ~100 layers (Section 2) but does not reproduce highway network experiments on ImageNet or CIFAR-10. A controlled comparison at matched depths and parameter counts would strengthen the claim that hardwired identity is superior to learned gating. The CIFAR-10 comparison in Table 6 includes highway network results from the literature, but these are at different network widths and training configurations, making direct comparison difficult.

  • No plain network experiments at ImageNet beyond 34 layers. Table 3 shows plain-34 at 28.54% top-1 error, and Table 3 compares against VGG-16 (a different architecture family) at 28.07%. The paper does not show how plain-50, plain-101, or plain-152 bottleneck networks would perform on ImageNet, making it impossible to see whether the residual formulation is necessary for deeper bottleneck architectures or whether plain bottleneck networks might also scale somewhat better than plain basic-block networks. The footnote stating that "the degradation problem of plain nets is also witnessed for the bottleneck designs" is the only mention, without supporting data.

  • The overfitting at 1202 layers is acknowledged but not explored. The 1202-layer result (7.93% vs. 6.43% for 110-layer) raises the question: at what depth, for CIFAR-10 with its 50k images, does overfitting begin to offset optimization gains? The paper sweeps n = {3, 5, 7, 9, 18, 200} but skips the range between n = 18 (110 layers) and n = 200 (1202 layers), leaving a large gap in understanding the depth-overfitting tradeoff curve.

  • No ablation of the ReLU-after-addition design choice. The paper places ReLU after the element-wise addition (Οƒ(F(x) + x)). An alternative would be to apply ReLU to the residual path before addition (F(x) = Wβ‚‚Οƒ(W₁x), then y = F(x) + x). The pre-addition ReLU would restrict the residual to be non-negative, potentially limiting representational capacity. The paper does not test this alternative, so the empirical justification for post-addition ReLU is absent.

  • Layer response analysis is limited. Figure 7 measures only standard deviation of layer outputs. Additional analyses that would strengthen the preconditioning argument: (a) showing that the residual path's output before addition has mean near zero (not just small variance), (b) tracking how response magnitudes evolve during training (do they start small and stay small, or start large and shrink?), (c) showing that residual blocks closer to the input learn larger modifications than blocks near the output (which would be consistent with the intuition that early layers do more feature extraction while later layers refine).

Overall, the experiments provide a clean, well-controlled demonstration of the core claims. The degradation problem is convincingly characterized and shown to be solved by residual learning across multiple datasets and depth scales. The main limitations are: (1) no confidence intervals on ImageNet results, (2) no direct comparison with highway networks in a controlled setting, (3) the layer response analysis is suggestive but not a rigorous causal test of the preconditioning hypothesis, and (4) the transition from optimization benefit to overfitting at extreme depths is acknowledged but not systematically mapped.

6. Limitations and Trade-offs

6.1 The Degradation Problem Has No Theoretical Explanation β€” Only Empirical Characterization

The assumption or constraint. The paper diagnoses and names the degradation problem β€” deeper plain networks show higher training error than shallower ones β€” and provides compelling empirical evidence that it is not caused by vanishing gradients, overfitting, or insufficient training time. However, the paper offers no theoretical account of why SGD fails to find identity-like solutions in deep nonlinear stacks. The authors are explicit about this gap:

"We conjecture that the deep plain nets may have exponentially low convergence rates, which impact the reducing of the training error. The reason for such optimization difficulties will be studied in the future."

This is a conjecture, not a proven claim. The residual formulation is presented as a practical solution to a phenomenon whose underlying mechanism remains unexplained.

The consequence. Without a theoretical understanding of why degradation occurs, practitioners cannot predict when residual learning will be necessary versus when plain networks will suffice. The paper's evidence suggests degradation becomes significant around 20–34 layers for the specific architectures tested, but it offers no framework for estimating the degradation threshold as a function of network width, dataset size, task complexity, or optimization hyperparameters. A practitioner designing a custom architecture for a new domain cannot answer: "at what depth do I need to switch to residual connections?" β€” they can only rely on heuristics derived from the ImageNet and CIFAR-10 regimes. Moreover, without a theoretical model, it is unclear whether other architectural choices (different activation functions, different normalization schemes, different optimizers) might also mitigate degradation, potentially obviating the need for residual connections entirely. The residual formulation might be sufficient to solve degradation without being necessary, and the paper provides no framework for determining necessity.

What evidence exists in the paper. The degradation phenomenon is extensively documented β€” Figure 1 (CIFAR-10), Figure 4 left (ImageNet), Figure 6 left (CIFAR-10 depth sweep) β€” and the paper verifies that gradients flow healthily with batch normalization (Section 4.1). However, these are all behavioral characterizations. The only mechanistic analysis is the layer response magnitude measurement in Figure 7, which shows that residual functions have smaller standard deviations, but this is a post-hoc observation about trained networks, not a predictive theory about optimization dynamics. The conjecture about "exponentially low convergence rates" is stated without formal definition or empirical estimation of the convergence rate.

Mitigation status. The paper does not attempt to develop or test a theoretical model. It explicitly defers this to future work in the quoted passage. Section 1 frames the contribution as an empirical solution: "We provide comprehensive empirical evidence showing that these residual networks are easier to optimize." The paper succeeds on its own terms β€” providing an architectural solution that works β€” but the "why" question remains open, which limits the predictive utility of the findings for architecture design beyond the tested configurations.

6.2 The Difficulty Estimation Step Is Not Integrated Into a Practical Deployment Pipeline

The assumption or constraint. This limitation is structural, not about something the paper fails to study, but about a necessary step that the paper's framework implies but never addresses. For a practitioner deploying a ResNet in a production system, they need to know: given my compute budget and task requirements, which ResNet depth should I use? The paper answers this implicitly through a table of results (deeper is better, with diminishing returns), but it provides no principled method for selecting depth short of training every candidate architecture and comparing. The design rules (double filters when halving spatial size; use bottlenecks for depths above 34) are qualitative heuristics, not a decision procedure. This is a limitation not of the idea but of the guidance for adoption: the paper demonstrates that depth is now possible but does not help the practitioner decide how much depth is enough.

The consequence. A practitioner adopting ResNets faces an architecture search problem: they must select depth (18, 34, 50, 101, 152, or something in between), block type (basic vs. bottleneck), width multiplier, and shortcut projection strategy. The paper's results show monotonic improvement with depth on ImageNet (Table 4) but diminishing returns β€” 101 β†’ 152 layers yields only 0.32 top-1 points on 10-crop testing (Table 3) and 0.49 points on multi-scale testing (Table 4). At some depth, the marginal accuracy gain will be outweighed by the increased training time, inference latency, and memory footprint, but the paper does not characterize this cost-benefit tradeoff quantitatively. Without such characterization, a practitioner cannot determine whether ResNet-152's 11.3 billion FLOPs per forward pass (vs. ResNet-50's 3.8 billion, roughly 3Γ— more) is justified for their specific accuracy requirements. The paper also does not explore whether depth can be traded off against width β€” could a shallower but wider ResNet match the accuracy of a deeper but narrower one at lower latency?

What evidence exists in the paper. The relevant data is spread across Table 1 (FLOPs and layer counts), Table 3 (10-crop errors for different depths), and Table 4 (multi-scale errors). For CIFAR-10, Table 6 provides parameter counts and errors, showing that the step from 110 layers (1.7M params, 6.43% error) to 1202 layers (19.4M params, 7.93% error) is actively harmful. The paper acknowledges this overfitting but attributes it to dataset size rather than providing a general principle for when to stop deepening. The CIFAR-10 experiments skip from n = 18 (110 layers) to n = 200 (1202 layers), leaving a gap where the optimal depth for this dataset likely lies.

Mitigation status. Not addressed. The paper's focus is on demonstrating that residual learning enables depth scaling, not on optimizing depth for a given task. This is a reasonable scoping choice for a foundational paper, but it means the practitioner inherits the burden of architecture selection. One piece of implicit guidance: the paper uses bottleneck blocks for depths β‰₯50 and basic blocks for ≀34, suggesting a threshold where computational efficiency concerns dominate. But this is a heuristic, not a systematic cost-benefit analysis.

6.3 Evaluation Is Confined to Convolutional Architectures for Computer Vision

The assumption or constraint. All experiments β€” ImageNet classification, CIFAR-10 analysis, PASCAL VOC and COCO detection, ImageNet localization β€” use convolutional neural networks for image-based tasks. The residual learning principle is formulated generically (Equation 1 uses notation applicable to any layer type, including fully-connected layers), but the paper never tests residual learning outside the context of 2D convolutional architectures for visual recognition. Section 1 claims that "the residual learning principle is generic, and we expect that it is applicable in other vision and non-vision problems," but this is stated as an expectation, not a demonstrated fact.

The consequence. A practitioner working in domains outside computer vision β€” natural language processing, speech recognition, reinforcement learning, graph neural networks, or any domain using non-convolutional architectures β€” cannot assume that residual learning will transfer effectively. There are plausible reasons why the benefit might be domain-specific:

  • Convolutional layers have strong spatial locality. The identity shortcut passes through the entire spatial feature map unchanged. In domains where layer computations are global (e.g., self-attention, fully-connected layers), the identity might be a less natural or less useful reference point for learning residuals.
  • The degradation problem may behave differently in other architectures. The paper conjectures that degradation stems from difficulty approximating identity mappings with multiple nonlinear layers. In architectures where layers are shallower (e.g., transformers with only a few layers before the residual connection in each block) or where nonlinearities are different (GELU, sigmoid, tanh), the degradation threshold might occur at different depths or might not occur at all within practical depth ranges.
  • The CIFAR-10 experiments use a different architecture family (simpler, no bottlenecks, fewer filters), confirming that residual learning works across architectural variants within computer vision, but this is still within the convolutional image processing domain.

The paper's contention that residual learning is generic is plausible β€” indeed, the subsequent adoption of residual connections in transformers (Vaswani et al., 2017) would later validate this β€” but the 2015 paper provides no direct evidence.

What evidence exists in the paper. None. The paper is entirely vision-focused. The CIFAR-10 experiments demonstrate generality across dataset scales (50k vs. 1.28M training images) and image resolutions (32Γ—32 vs. 224Γ—224), but this is within the same modality (natural images) and same architecture class (convolutional networks). The detection results (Tables 7–11) show transfer to different tasks (classification β†’ detection β†’ localization) but still within the visual domain and still using convolutional backbones.

Mitigation status. The paper acknowledges this as a limitation of scope, not of principle. The claim of generality is explicitly hedged as an expectation: "we expect that it is applicable in other vision and non-vision problems." The paper does not attempt to test this expectation. For a practitioner in non-vision domains, this means the residual learning principle must be validated independently on their architecture and task of interest. Historical precedent would eventually confirm the paper's optimism, but at the time of publication, this was a genuine uncertainty.

6.4 No Comparison Against Ensembles of Shallower Networks at Equal Parameter Count

The assumption or constraint. The paper's core argument is that residual learning enables depth to be effectively exploited β€” deeper ResNets achieve higher accuracy than shallower ResNets, and this is attributed to the quality of learned representations at greater depth. However, a natural alternative explanation is that deeper networks simply have more parameters, and the accuracy gains come from increased capacity rather than increased depth per se. The paper partially controls for this by comparing plain and residual networks of identical depth and parameter count (Tables 2, 6), but it does not compare a single deep ResNet against an ensemble of shallower networks with the same total parameter count. For example, an ensemble of three 34-layer ResNets (each with ~3.6 billion FLOPs) would have comparable total computational cost to a single 101-layer ResNet (7.6 billion FLOPs for two 34-layer ensembles, or a 50-layer at 3.8 billion plus a 34-layer at 3.6 billion for a total near the 101-layer's 7.6 billion). Does the 101-layer single model outperform the ensemble of shallower models? If not, the "depth benefit" might actually be an "ensemble benefit" achievable through other means.

The consequence. Without this comparison, the paper cannot rule out the hypothesis that the accuracy gains from deeper ResNets are primarily due to increased parameter count rather than the representational benefits of hierarchical depth. This is particularly important because ensembles are a well-known, simple method for improving accuracy that is orthogonal to architecture design. If an ensemble of shallow ResNets matches the accuracy of a deep ResNet at equal computational cost, the practical recommendation might be "use ensembles of modestly deep ResNets" rather than "train extremely deep ResNets." The value proposition of depth over alternative ways to spend parameter and compute budgets would be unclear.

The paper does implicitly address a weaker version of this concern: by comparing plain and residual networks at exactly matched parameter counts (Tables 2, 3, 6), it shows that residual connections provide benefits beyond what additional parameters alone would achieve (since the plain networks have the same number of parameters but worse performance). However, this does not address the ensemble alternative β€” ensembles use parameters differently (multiple independent models) rather than stacking them in depth.

What evidence exists in the paper. The paper does provide ensemble results, but only for the final competition submission: the ILSVRC 2015 winning entry ensembles six models of different depths (Table 5). The single-model 152-layer ResNet achieves 4.49% top-5 error (Table 4), while the ensemble achieves 3.57% (Table 5). This shows that ensembles further improve upon single deep models, but it does not answer the reverse question: can ensembles of shallower models match the single deep model? The only relevant data point is indirect: VGG-16 and VGG-19 were typically used in ensembles (VGG's ensemble achieved 6.8% top-5 test error, Table 5), while ResNet-152 single-model achieves 4.49% β€” clearly the deep ResNet single model outperforms VGG ensembles. But this compares different architecture families (ResNet vs. VGG), not shallow ResNet ensembles vs. deep single ResNets.

Mitigation status. Not addressed. The paper's focus is on demonstrating that residual learning solves the degradation problem specifically β€” enabling a single network to benefit from depth without optimization collapse. The ensemble alternative is a different axis (inference-time computation via multiple models) that the paper does not engage with. This is a fair scoping decision given the paper's goal, but it means the claim that depth is the most efficient way to use parameters remains unverified against the ensemble baseline.

6.5 Training Hyperparameters Are Not Tuned for Depth β€” Only One Configuration Is Tested

The assumption or constraint. Throughout the ImageNet experiments, all network depths (18, 34, 50, 101, 152) are trained with identical hyperparameters: initial learning rate 0.1, weight decay 0.0001, momentum 0.9, mini-batch size 256, and the same learning rate decay schedule (divide by 10 at error plateaus, up to 60 Γ— 10^4 iterations). These hyperparameters were presumably optimized for the shallower networks or inherited from prior work (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015), but the paper provides no evidence that they are optimal β€” or even near-optimal β€” for the deeper variants. A single configuration is used for all depths, and no hyperparameter sensitivity analysis is presented.

For CIFAR-10, the 110-layer network requires an ad-hoc learning rate warmup (0.01 for ~400 iterations before switching to 0.1) to stabilize early training, acknowledging that the standard hyperparameters are not universally appropriate. Yet even then, no systematic exploration of learning rate schedules, weight decay values, or optimization algorithms for different depths is conducted.

The consequence. The paper's headline comparison β€” deeper ResNets achieve lower error than shallower ones β€” could be confounded by suboptimal hyperparameters for the shallower variants. If, for example, the 18-layer ResNet would benefit from a different learning rate schedule than the 152-layer ResNet, but both are evaluated under the 152-layer's implicitly tuned schedule, the 18-layer result may understate what shallower networks can achieve. Conversely, if the 152-layer ResNet could achieve even better accuracy with depth-specific hyperparameter tuning, the paper's reported accuracy gains from depth are lower bounds β€” but the fair comparison between depths becomes ambiguous.

More subtly, the degradation problem in plain networks might be partially addressed by depth-aware hyperparameter tuning. The paper rules out "more iterations" as a solution (Section 4.1 footnote), but it does not test alternative learning rate schedules, warmup strategies, or optimization algorithms (e.g., Adam, RMSProp) for plain networks. It remains possible that plain-34, with appropriate hyperparameter tuning, could match or approach plain-18's training error β€” which would weaken the argument that an architectural solution (residual connections) is necessary to address degradation.

What evidence exists in the paper. The CIFAR-10 110-layer warmup (Section 4.2) is the paper's own evidence that depth interacts with optimization hyperparameters: "we find that the initial learning rate of 0.1 is slightly too large to start converging." This is an informal, post-hoc adjustment rather than a systematic study, but it reveals that hyperparameter sensitivity increases with depth. The footnote adds that without warmup, "it starts converging (<90% error) after several epochs, but still reaches similar accuracy" β€” suggesting the warmup is a convenience rather than a correctness requirement, but the eventual accuracy being "similar" is not quantified.

For ImageNet, no hyperparameter tuning across depths is reported. The learning rate schedule (divide by 10 at error plateaus, up to 600k iterations) is the same for all depths. There is no evidence that this schedule was validated as appropriate for, say, ResNet-152 versus ResNet-18.

Mitigation status. Partially addressed for CIFAR-10 (the 110-layer warmup), not addressed for ImageNet. The paper's position is implicit: the identical hyperparameters across depths make the comparison fair (all depths get the same treatment), even if they are not optimal for each depth. This is a defensible methodological choice β€” tuning per depth would introduce a confound (is the improvement from depth or from better hyperparameters?) β€” but it means the reported accuracy numbers may not reflect the best achievable performance at each depth, and the claimed benefit of depth may be partially attributable to hyperparameters that happen to favor deeper architectures. The paper does not discuss this tradeoff.

6.6 The Scaling Behavior of Residual Learning β€” When Depth Stops Helping β€” Is Not Characterized

The assumption or constraint. The paper demonstrates that residual learning enables accuracy gains from increased depth up to 152 layers on ImageNet (Tables 3–4), with monotonic improvement across 18, 34, 50, 101, and 152 layers. However, the paper provides no characterization of when this trend stops or reverses. On CIFAR-10, the 1202-layer ResNet performs worse than the 110-layer ResNet (7.93% vs. 6.43%, Table 6), but this is attributed to overfitting due to the small dataset (50k images) rather than an inherent depth limitation: "The 1202-layer network may be unnecessarily large (19.4M) for this small dataset." For ImageNet, with its 1.28 million training images and 1000 classes, the paper does not test depths beyond 152 layers, leaving open the question: does ResNet accuracy continue to improve at 200, 300, or 500 layers on a sufficiently large dataset, or does residual learning itself eventually encounter a degradation-like phenomenon?

The consequence. The paper's central promise is that residual learning solves the degradation problem β€” deeper networks can be optimized and do yield better accuracy. But the paper establishes this only for depths up to 152 on ImageNet and 110 on CIFAR-10 (with 1202 overfitting, not optimization-failing). A practitioner considering whether to invest in training a 300-layer or 500-layer ResNet for a large-scale dataset has no evidence about whether the trend of diminishing returns continues linearly, saturates, or reverses. The 1202-layer CIFAR-10 result raises the possibility that, even on large datasets, there exists some depth beyond which additional layers provide zero or negative benefit β€” either due to overfitting or to a deeper form of optimization difficulty that residual learning only postpones rather than eliminates.

Furthermore, the transition from "depth helps" to "depth overfits" is not mapped. On CIFAR-10, the gap between n = 18 (110 layers, 6.43%) and n = 200 (1202 layers, 7.93%) is large and unexplored. At what depth does overfitting begin to dominate for CIFAR-10? Does the optimal depth scale predictably with dataset size? These questions are central to practical deployment but are not addressed.

What evidence exists in the paper. The only evidence about the limits of residual learning's effectiveness is the 1202-layer CIFAR-10 result (Figure 6 right, Table 6). This single data point shows that training remains possible (optimization does not collapse β€” training error reaches <0.1%) but generalization degrades. The paper's diagnostic (overfitting, not degradation) is plausible β€” the training error is near zero while test error is elevated β€” but this is an inference, not a controlled demonstration. The paper does not, for instance, show that the 1202-layer network would outperform the 110-layer network on a larger dataset, which would confirm that overfitting (not some other depth-related pathology) is the cause of the worse test performance.

For ImageNet, there is no experiment probing the depth limit. The 152-layer ResNet is the deepest tested, and it still shows improvement over 101 layers (21.75% β†’ 21.43% top-1 on 10-crop, Table 3; 19.87% β†’ 19.38% top-1 on multi-scale, Table 4). The marginal gain is shrinking (0.32 points from 101 to 152 vs. 1.10 points from 50 to 101), suggesting diminishing returns, but whether the curve plateaus, asymptotes, or eventually turns upward is unknown.

Mitigation status. Not addressed. The paper's scope is to demonstrate that residual learning makes deep networks trainable and beneficial up to depths far beyond prior work. Characterizing the ultimate limits of depth scaling β€” whether residual learning fundamentally eliminates degradation or merely postpones it to depths beyond practical reach β€” is beyond the paper's empirical scope. The authors do not claim to have found the depth ceiling; the 1202-layer experiment is presented as an exploration ("Exploring Over 1000 layers"), not an attempt to locate the failure point. For practitioners, this means that the paper provides strong evidence for depths up to ~150 layers on large-scale image classification, but no guidance for determining whether deeper networks would help or hurt on their specific task and dataset scale.

7. Implications and Future Directions

How This Work Changes the Landscape

The ResNet paper does not merely introduce a new architecture β€” it redefines the goal of neural network design from "what functions can be represented" to "what functions can be optimized." Before this work, the dominant narrative held that depth was limited by vanishing/exploding gradients, and that solving gradient propagation (via careful initialization and batch normalization) would unlock arbitrarily deep networks. The paper's diagnostic is different and more fundamental: even when gradients flow healthily, SGD cannot find good solutions in deep plain networks because the optimization landscape itself is ill-conditioned. The degradation problem (Figure 1) is not a gradient problem β€” it is a landscape problem.

This reframing shifts the research agenda in a specific way. Instead of asking "how do we get gradients to flow better?" β€” a question that normalized initialization and batch normalization had largely answered β€” the field begins asking "how do we structure the parameter space so that SGD can navigate it effectively?" The residual formulation answers this by making the default behavior (near-zero weights β†’ near-identity function) correspond to a good solution, so the optimizer starts in a favorable region and only needs to learn perturbations. This is a preconditioning argument, not a capacity argument, and it changes what architectural innovation means: the goal is no longer just to increase representational power but to shape the optimization landscape to be friendly to first-order methods.

The paper resolves a specific contradiction in the prior literature. VGG nets (Simonyan and Zisserman, 2015) had shown that 16–19 layer networks benefit from depth, but attempts to go deeper produced diminishing or negative returns. Highway networks (Srivastava et al., 2015) introduced learned gating to facilitate information flow, yet "have not demonstrated accuracy gains with extremely increased depth (e.g., over 100 layers)." The ResNet paper reconciles these observations: depth is beneficial, but only when the architecture provides a structural bias toward identity mappings that keeps the optimization problem well-conditioned. Learned gating allows the network to discover identity-like behavior but does not force the optimizer to stay near identity β€” and the optimizer, left to its own devices in a deep plain network, wanders into regions where convergence is exponentially slow. Hardwired identity shortcuts eliminate this possibility.

The paper also establishes depth as a reliably scalable dimension of architecture design, analogous to how width had been scaled previously. Before ResNet, practitioners could increase width (more filters per layer) with predictable accuracy gains, but increasing depth was a gamble β€” sometimes it helped, sometimes it hurt, and the conditions were poorly understood. After ResNet, depth becomes a safe knob to turn: Tables 3–4 show monotonic improvement from 18 to 152 layers on ImageNet, and Table 6 shows monotonic improvement from 20 to 110 layers on CIFAR-10. This reliability transforms depth from a source of anxiety into a standard engineering lever.

Perhaps most significantly, the paper demonstrates that a single, simple design principle β€” learn residuals with reference to identity β€” generalizes across tasks, scales, and architectures. The same basic building block (Figure 2) works for ImageNet classification (152 layers, 1000 classes, 1.28M images), CIFAR-10 analysis (110 and 1202 layers, 10 classes, 50k images), PASCAL VOC detection, and COCO detection and segmentation. The COCO result is particularly revealing: replacing VGG-16 with ResNet-101 in an otherwise identical Faster R-CNN pipeline yields a 28% relative improvement in mAP@[.5, .95] (Table 8), solely from better learned representations. This is not a task-specific trick β€” it is evidence that residual learning produces fundamentally better features that transfer across visual recognition problems.

Research directions that become more attractive after this paper:

  • Architecture as optimization preconditioning. The paper's central insight β€” that architectural choices can be understood as shaping the optimization landscape β€” opens a new lens for designing networks. Future architectures can be evaluated not just by their representational capacity but by how they condition the loss surface for SGD. The layer response analysis in Figure 7 (showing that residual functions have genuinely smaller magnitudes) provides a template for this kind of mechanistic validation.
  • Extreme depth as a practical tool. With the degradation problem solved, depth becomes a viable dimension for scaling model capacity. The paper's 152-layer ImageNet model and 1202-layer CIFAR-10 model demonstrate that depths previously considered absurd are now achievable, inviting exploration of even deeper networks on larger datasets.
  • Residual representations for transfer learning. The detection and localization results (Tables 7–14) show that residual representations transfer exceptionally well, suggesting that ResNet-pretrained features might become a standard backbone for diverse vision tasks β€” which indeed happened in subsequent years.

Research directions that become less urgent:

  • Gated shortcut mechanisms. The paper's comparison with highway networks β€” and the finding that hardwired identity outperforms learned gating, especially at extreme depths β€” suggests that complex gating mechanisms for information flow are unnecessary. The simplicity of identity shortcuts is not a limitation but a feature.
  • Vanishing gradient solutions for very deep networks. The paper explicitly verifies that gradients flow healthily in plain networks with batch normalization (Section 4.1), yet degradation persists. This redirects attention away from gradient magnitude and toward optimization landscape geometry. Further refinements to gradient normalization (beyond BN) may provide diminishing returns compared to architectural preconditioning.

Follow-Up Research This Work Enables

Formal theory of the degradation problem: exponential convergence rates in deep plain networks. The paper conjectures that "deep plain nets may have exponentially low convergence rates" but provides no formal analysis. A follow-up could construct a toy setting β€” e.g., deep linear networks or deep networks with ReLU activations on synthetic data β€” and prove that SGD convergence time grows exponentially with depth when the target function is identity (or near-identity). The analysis would characterize the condition number of the Hessian at the identity solution as a function of depth, width, and initialization scale. A strong result would show that the Hessian becomes increasingly ill-conditioned with depth in plain networks but remains well-conditioned in residual networks, providing a theoretical explanation for Figure 7 (smaller residual responses correlate with better conditioning). This would transform the paper's empirical conjecture into a predictive theory, enabling quantitative predictions about when residual connections are necessary based on network depth and width.

Systematic mapping of the depth–overfitting tradeoff on large-scale datasets. The paper's CIFAR-10 experiments show that 1202 layers overfit (7.93% test error vs. 6.43% for 110 layers), but the depth range between 110 and 1202 layers is unexplored, and ImageNet experiments stop at 152 layers without detecting an overfitting threshold. A follow-up could train ResNets at 200, 300, 500, and 800 layers on ImageNet (or a comparably large dataset) to locate the depth at which overfitting begins to offset optimization gains. The experiment would track both training and validation error as depth increases, distinguishing between genuine overfitting (validation error rises while training error falls) and a new form of degradation (training error itself plateaus or rises). A key measurement: does the optimal depth scale predictably with dataset size? If 152 layers is near-optimal for 1.28M images, does 300 layers become optimal for 10M images? This would provide the practical depth-selection guidance that the paper currently lacks.

Direct controlled comparison of identity shortcuts vs. learned gating at matched depth and parameter count on ImageNet. The paper critiques highway networks for not scaling beyond ~100 layers but does not reproduce highway experiments in a controlled setting. A follow-up would implement highway networks using the exact same convolutional backbone, depth, width, training hyperparameters, and data augmentation as ResNet-50/101/152, with learned gating replacing identity shortcuts. The key measurement: at what depth does the highway network's accuracy diverge from ResNet accuracy? If highway-50 matches ResNet-50 but highway-101 underperforms ResNet-101, this would pinpoint where learned gating fails and hardwired identity becomes essential. Additionally, analyzing the learned gate values β€” do they saturate toward identity or diverge? β€” would reveal whether the optimization difficulty is that gates fail to learn identity-like behavior or that identity-like gates are learned but still provide weaker optimization conditioning than hardwired shortcuts.

Residual learning in non-convolutional architectures: testing the generality claim. The paper asserts that "the residual learning principle is generic," but all experiments use 2D convolutional networks for vision. A follow-up would test residual connections in a fundamentally different architecture class: fully-connected networks for tabular data, recurrent networks for sequence modeling, or early attention-based architectures. A clean experiment would replicate the degradation phenomenon in a non-convolutional setting (e.g., train plain fully-connected networks of increasing depth on a moderate-scale supervised learning task, showing that training error increases with depth beyond some threshold) and then demonstrate that adding identity shortcuts between every few fully-connected layers reverses the degradation. A negative result β€” residual connections providing no benefit in fully-connected or recurrent architectures β€” would bound the generality claim and suggest that residual learning interacts specifically with the local connectivity and weight sharing of convolutional layers.

The role of the ReLU-after-addition design choice: what if ReLU is applied before the shortcut? The paper places ReLU after element-wise addition ($\sigma(\mathcal{F}(x) + x)$). An alternative β€” apply ReLU only to the residual path, then add to the shortcut ($\mathcal{F}(x) = W_2 \sigma(W_1 x)$, then $y = \mathcal{F}(x) + x$) β€” would restrict the residual to be non-negative, since ReLU outputs are β‰₯ 0. This means the block output could only be greater than or equal to the input (element-wise), eliminating the ability to learn negative corrections. A follow-up could train ResNet-34 and ResNet-50 variants with pre-addition ReLU on ImageNet, measuring: (a) does accuracy degrade, and by how much?, and (b) do the learned residual functions exhibit different statistics (mean, variance, sparsity) compared to the post-addition ReLU design? If pre-addition ReLU performs substantially worse, this confirms that the ability to learn both positive and negative perturbations from identity is essential β€” residual learning works not just because identity is a good reference point, but because the residual correction can freely adjust the representation in any direction. If pre-addition ReLU performs surprisingly well, it would suggest that the residual bias is toward larger activations (positive corrections only), which would revise the preconditioning interpretation.

Can depth be traded off against width, and what is the compute-optimal architecture? The paper scales depth while keeping width fixed per stage (e.g., ResNet-152 uses the same filter counts as ResNet-50, with more blocks per stage). A follow-up could perform a controlled FLOPs-matched comparison: for a fixed computational budget (e.g., 11.3 billion FLOPs, matching ResNet-152), sweep over depth–width combinations (e.g., shallower but wider: ResNet-50 with doubled filters vs. ResNet-101 with 1.5Γ— filters vs. ResNet-152 with standard width). The experiment would identify the Pareto frontier of depth vs. width at constant compute, answering whether the paper's implicit strategy (favor depth over width) is compute-optimal or whether a more balanced scaling approach yields better accuracy. This directly extends the paper's methodology β€” FLOPs-matched comparison between plain and residual networks β€” to the question of architectural resource allocation within the residual family.

Practical Applications and Downstream Use Cases

Large-scale image classification and retrieval in production. For any organization serving image classification at scale (photo organization, content moderation, product recognition), the paper provides an immediate upgrade path: replace VGG-16/19 backbones with ResNet-101 or ResNet-152. The 152-layer ResNet achieves 4.49% top-5 single-model error on ImageNet validation (Table 4), outperforming all prior ensemble results, while requiring only 11.3 billion FLOPs β€” less than VGG-19 at 19.6 billion FLOPs and VGG-16 at 15.3 billion FLOPs (Table 1). This means a deployed ResNet-152 provides better accuracy than VGG ensembles at lower computational cost per query, reducing both latency and infrastructure expenses. The fact that the same architecture family spans from ResNet-18 (1.8 billion FLOPs, 27.88% top-1 error) to ResNet-152 (11.3 billion FLOPs, 21.43% top-1 error on 10-crop) means practitioners can select a point on the accuracy–compute curve that matches their deployment constraints.

Object detection and instance segmentation in autonomous systems. The COCO detection results (Tables 8–9) demonstrate that swapping VGG-16 for ResNet-101 in Faster R-CNN improves mAP@[.5, .95] from 21.2% to 27.2% β€” a 28% relative gain β€” with no changes to the detection algorithm itself. The additional improvements (box refinement, context, multi-scale testing) push single-model performance to 34.9% mAP@[.5, .95] and ensemble performance to 37.4% (Table 9). For autonomous driving, robotics, or surveillance systems where detection and segmentation accuracy directly impact safety and reliability, a 28% relative improvement in the core metric from a backbone swap is a massive practical gain. The detection improvements are "solely contributed by the features learned by the better network" (Appendix A), meaning the benefit comes from better pretrained representations β€” practitioners can integrate ResNet backbones into existing detection pipelines with minimal engineering effort.

Medical image analysis and other domains with limited labeled data. The paper demonstrates that residual representations transfer effectively from ImageNet classification to detection and localization (Tables 7–14) without task-specific architectural modifications. For domains like medical imaging, where labeled training data is scarce and pretraining on ImageNet is standard practice, switching from VGG-pretrained to ResNet-pretrained backbones should yield accuracy improvements comparable to the detection gains in the paper β€” simply because the transferred features are of higher quality. The CIFAR-10 results (Table 6) are particularly relevant: the 110-layer ResNet achieves 6.43% error with only 1.7M parameters, and the 20-layer ResNet achieves 8.75% with 0.27M parameters. These compact, high-accuracy models demonstrate that residual learning produces efficient representations even at modest parameter counts β€” suitable for deployment in resource-constrained medical imaging pipelines where GPU memory and inference time are limited.

Competition-level computer vision systems. The paper's sweep of ILSVRC and COCO 2015 competition wins β€” classification, detection, localization, COCO detection, and COCO segmentation β€” establishes ResNets as the default backbone for pushing state-of-the-art on visual recognition benchmarks. A team entering a vision competition in 2016 or later can adopt ResNet-101 or ResNet-152 pretrained on ImageNet as their starting point, with confidence that the backbone is competitive and the representations transfer. The paper's specific recipe β€” pretrain on ImageNet classification, fine-tune on target task with BN statistics frozen (Appendix A) β€” provides an immediately replicable workflow. The 28% relative COCO improvement and the 64% relative localization error reduction (Tables 13–14) quantify the expected gains from this backbone swap.

When to Prefer This Method

The paper positions residual networks against two named alternatives β€” plain networks (same architecture without shortcuts) and highway networks (learned gating) β€” and the tradeoff is clearly articulated through empirical results:

  • Prefer ResNets over plain networks when depth exceeds ~20 layers. The degradation problem becomes visible at 34 layers on ImageNet (plain-34: 28.54% top-1 error vs. plain-18: 27.94%, Table 2) and at 32+ layers on CIFAR-10 (Figure 6 left). For networks shallower than ~20 layers, plain and residual networks perform comparably (18-layer: 27.94% plain vs. 27.88% ResNet, Table 2), though ResNets converge faster (Figure 4). The break-even point depends on task complexity and dataset size, but the paper's evidence suggests that residual connections provide non-trivial optimization benefits for any architecture deep enough that the degradation problem could plausibly arise.

  • Prefer ResNet identity shortcuts over highway network gated shortcuts when targeting extreme depth (100+ layers). Highway networks have not demonstrated accuracy gains beyond ~100 layers (Section 2), while ResNets successfully train at 152 layers on ImageNet and 1202 layers on CIFAR-10. The paper attributes this to the hardwired identity path β€” it cannot be closed by learned gates, so the optimizer is structurally prevented from abandoning the identity reference. If the target depth is modest (tens of layers), both approaches may work, but for pushing depth to its limits, identity shortcuts are the demonstrated choice.

  • Prefer zero-padding shortcuts (option A) over projection shortcuts (options B, C) when parameter count parity with plain baselines is required. Table 3 shows that zero-padding (25.03% top-1) provides most of the benefit over plain networks (28.54%), with projection shortcuts adding 0.51–0.84 additional percentage points at the cost of extra parameters. For controlled scientific comparisons where parameter count must be identical, option A is sufficient to demonstrate the residual learning effect. For deployment, option B (projections only for dimension changes) provides a modest accuracy improvement at negligible additional cost.

  • Prefer bottleneck blocks over basic blocks for depths β‰₯50 layers to maintain computational tractability. The paper states that deeper non-bottleneck ResNets "also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets" (Section 4.1 footnote). ResNet-152 with bottleneck blocks requires 11.3 billion FLOPs β€” less than VGG-16 at 15.3 billion β€” whereas a hypothetical 152-layer basic-block ResNet would be substantially more expensive. The bottleneck design is an engineering choice, not a requirement for residual learning, but it is essential for making extreme depths practical.