Year in Review: 2019 Highlights from the Uber Engineering Blog

ArXiv: 1803.03635

🎯 Pitch

A randomly initialized neural network already contains a tiny subnetwork that can train in isolation to match the full model’s accuracy—sometimes even faster—but the catch is you must reuse the original initial weights, not reinitialize randomly. Pruning alone isn’t enough; the specific initialization values are the real "winning ticket."

---

1. Executive Summary

This paper empirically studies whether dense, randomly-initialized neural networks contain sparse subnetworks that can train from scratch and match the full network's test accuracy. Using the Lenet architecture on MNIST and Conv-2/4/6, VGG-19, and Resnet-18 architectures on CIFAR10, the authors articulate the lottery ticket hypothesis — the claim that any dense feed-forward network contains a subnetwork ("winning ticket") whose original initialization enables it to train in isolation to comparable accuracy in at most the same number of iterations as the full network. Winning tickets are identified via iterative magnitude-based pruning followed by resetting surviving weights to their initial values (rather than retaining trained values). The winning tickets are consistently 10–20% of the original network's size, often learning faster and generalizing better than the full model — a 3.5× speedup in early-stopping time and up to 3.5 percentage points higher test accuracy — establishing that initialization, not just architecture, is the key ingredient, as randomly reinitializing the same sparse structure destroys these improvements.

2. Context and Motivation

The Core Puzzle: Why Train Big Networks if We Can Prune Them Small?

The paper opens with a question that, on its face, seems to contradict standard deep learning practice. Neural network pruning techniques—methods that remove unnecessary weights from trained networks—can eliminate over 90% of parameters without harming accuracy (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015). These pruned networks are smaller, faster at inference, and consume less energy. But if a network can be reduced in size after training, why not simply train the smaller architecture from the beginning? Doing so would make training itself more efficient—less memory, fewer FLOPs, faster iteration cycles.

The problem is that contemporary experience at the time of this paper strongly suggests this doesn't work. The architectures uncovered by pruning are, in the words of practitioners, "harder to train from the start, reaching lower accuracy than the original networks." Two prominent examples of this conventional wisdom appear in the paper's citations:

• Li et al. (2016): "Training a pruned model from scratch performs worse than retraining a pruned model, which may indicate the difficulty of training a network with a small capacity."
• Han et al. (2015): "During retraining, it is better to retain the weights from the initial training phase for the connections that survived pruning than it is to re-initialize the pruned layers...gradient descent is able to find a good solution when the network is initially trained, but not after re-initializing some layers and retraining them."

The standard pruning pipeline thus works as follows: train a large network → prune away unimportant connections → fine-tune the surviving weights. Crucially, fine-tuning starts from the trained values of the surviving weights, not from random initialization. This pipeline implicitly assumes that the initial training serves two purposes: (1) identifying which connections matter (the mask), and (2) setting the surviving weights to values from which they can be successfully fine-tuned. The second purpose is what makes training the sparse network from scratch fail—if you take the same architecture but randomly initialize it, gradient descent cannot find a comparably good solution.

This paper zeroes in on the fundamental question behind this phenomenon: Is there something special about the initialization of the surviving weights in a pruned network, beyond just the architecture they form? The authors' motivating insight is that pruning might not just select which weights to keep—it might also identify weights whose original, pre-training values were already well-suited for learning. If true, this would mean that the dense network contains, at initialization, a sparse subnetwork that is already "configured" to learn effectively, and that training the full network is merely a mechanism for discovering this subnetwork.

Why This Matters: Training Efficiency and Theoretical Understanding

The paper's motivating question touches on two distinct but equally important concerns.

Practical: can we make training cheaper? If small, trainable subnetworks exist within large networks from the moment of initialization, then one could imagine a training procedure that identifies and trains only the winning ticket—skipping the expensive process of training the full dense network entirely. This would reduce the computational cost of training, which was (and remains) a primary bottleneck in deep learning research and deployment. Training a neural network once is expensive; identifying a winning ticket via the paper's iterative pruning method requires training it 15+ times. But if the phenomenon is real and general, it opens the door to more efficient discovery mechanisms that could make training cheaper overall.

Theoretical: what does overparameterization actually do? The paper connects to a deeper theoretical question that was gaining attention in 2018–2019: why are overparameterized neural networks easier to optimize? This is counterintuitive from an optimization perspective—you would expect more parameters to mean a harder optimization problem. Yet the evidence was clear that larger networks train faster and generalize better. Prior theoretical work (e.g., Du et al., 2019, proving that sufficiently overparameterized two-layer ReLU networks trained with SGD converge to global optima) had begun to offer explanations grounded in the optimization landscape. The lottery ticket hypothesis offers a complementary, mechanistic explanation: larger networks are easier to train because they contain more possible subnetworks, increasing the odds that at least one of them is a "winning ticket" with a fortuitous initialization. By this logic, overparameterization is not about making the optimization landscape smoother—it's about providing a larger pool of subnetworks from which SGD can implicitly select.

This connects to a related observation about the relationship between pruning and generalization. The paper invokes the concept of an "Occam's Hill" (Rasmussen & Ghahramani, 2001), where moderately compressed models generalize better than both severely overparameterized models (which overfit) and severely compressed models (which underfit). Recent theoretical work at the time had shown tighter generalization bounds for networks that could be compressed further (Zhou et al., 2018, for pruning/quantization; Arora et al., 2018, for noise robustness). The lottery ticket hypothesis suggests that larger networks might explicitly contain simpler representations in the form of winning tickets, offering a concrete mechanism for why compression and generalization are linked.

Prior Approaches and Where They Fall Short

The paper situates its contribution against several research directions that aimed to train small or sparse networks. Each falls short in a way that the lottery ticket hypothesis addresses.

Pruning before training. Prior work attempted to design small architectures from scratch: SqueezeNet (Iandola et al., 2016) and MobileNets (Howard et al., 2017) engineered compact image-recognition architectures an order of magnitude smaller than standard ones. Denil et al. (2013) represented weight matrices as low-rank factorizations. Li et al. (2018) restricted optimization to a small, randomly-sampled subspace of the parameter space and successfully trained networks under this restriction. These approaches demonstrate that small networks can be designed and trained, but they do not explain why the specific sparse architectures uncovered by pruning seem uniquely difficult to train from scratch—they rely on clever architectural design rather than explaining the pruning phenomenon.

Pruning after training (the standard paradigm). Most relevant to this paper is the extensive literature on compressing trained networks. LeCun et al. (1990) and Hassibi & Stork (1993) pioneered pruning based on second-order derivatives. Han et al. (2015) showed that simple magnitude-based pruning could dramatically reduce network size while preserving accuracy. Subsequent work refined this: Guo et al. (2016) dynamically restored pruned connections that became relevant again; Han et al. (2017) and Jin et al. (2016) used dense-sparse-dense training flows; various authors proposed pruning heuristics based on activations (Hu et al., 2016), redundancy (Mariet & Sra, 2016), or energy efficiency (Yang et al., 2017). Structured pruning methods removed entire filters (Li et al., 2016) or channels (He et al., 2017) for hardware efficiency.

All of these approaches share a common feature: the pruned network is fine-tuned starting from the trained weights of the surviving connections. They do not address whether the sparse architecture could have been trained from scratch. The lottery ticket hypothesis asks precisely the question these methods avoid: what if you reset those weights to their initial values rather than retaining their trained values?

Training with sparsity from the start. Several contemporaneous methods explored training sparse networks without a dense pre-training phase. Bellec et al. (2018) used "deep rewiring," randomly reinitializing weights that reached zero during sparse training. Srinivas et al. (2017) and Louizos et al. (2018) learned gating variables that minimized non-zero parameter counts during training. Bayesian perspectives on dropout (Gal et al., 2017; Kingma et al., 2015; Molchanov et al., 2017) learned per-weight or per-unit dropout probabilities, naturally sparsifying networks as some dropout probabilities reached 1. Narang et al. (2017) integrated magnitude-based pruning directly into training.

These methods show that sparse networks can be trained without a dense pre-training phase if sparsity is introduced gradually or guided by learned parameters. However, they do not address the specific phenomenon the lottery ticket hypothesis highlights: that within a randomly initialized dense network, there already exists a sparse subnetwork that, without any special training tricks, can be trained from scratch to match the full network. The lottery ticket hypothesis is about the existence of a pre-existing, fortuitously-initialized subnetwork—not about clever training schemes for sparse architectures.

The Direct Conflict with Liu et al. (2019)

Perhaps the most important piece of context for this paper is its relationship to Liu et al. (2019), "Rethinking the Value of Network Pruning," which was published contemporaneously and reaches seemingly contradictory conclusions. Liu et al. found that pruned architectures, when randomly reinitialized and trained from scratch, could match the accuracy of the original network—directly challenging the conventional wisdom that pruned architectures are inherently hard to train. On VGG-19 for CIFAR10, they found that networks pruned by up to 80% and randomly reinitialized could match the original accuracy.

This paper does not dispute Liu et al.'s findings but rather refines them. The authors confirm in their own experiments (Section 4, Figure 7) that at moderate sparsity levels, randomly reinitialized pruned networks can indeed match the original accuracy—reproducing Liu et al.'s result. However, they show that when pruning is pushed beyond this point (to 98.5% sparsity for VGG-19), the network only maintains accuracy with its original initialization—random reinitialization fails dramatically. The lottery ticket hypothesis thus proposes a boundary condition on Liu et al.'s finding: highly overparameterized networks can absorb substantial pruning regardless of initialization, but beyond a certain threshold, the specific initialization becomes essential.

This resolution is significant because it reframes the debate. Liu et al. questioned whether pruning was really about identifying important architectures, since random reinitialization worked. The lottery ticket hypothesis answers: at moderate sparsity, architecture alone may suffice (the structure encodes a useful inductive bias); at extreme sparsity, initialization becomes the critical factor—the winning ticket has "won the initialization lottery."

How This Paper Positions Itself

The paper frames its contribution not as proposing a new pruning method or a new training algorithm, but as articulating and empirically validating a hypothesis about the nature of neural networks. This is an unusual positioning for a systems/empirical paper—it's closer to a scientific discovery claim than a method claim. The central assertion is that winning tickets exist as a property of dense, randomly-initialized networks, independent of any particular training scheme.

The paper explicitly distinguishes two components of a winning ticket: structure (the mask $m$—which connections are present) and initialization (the values $m \odot \theta_0$ assigned to those connections). The random reinitialization experiments (where the mask $m$ is preserved but $\theta_0$ is resampled) isolate the contribution of initialization: if the reinitialized network performs poorly, the initialization was crucial. This is the paper's key experimental design choice and the basis for claiming that winning tickets have "won the initialization lottery."

The paper also positions its iterative pruning procedure as a discovery method, not a training method. The goal is not to propose iterative magnitude pruning as the best way to train networks—it's to use it as a tool to find winning tickets and study their properties. This is important context for understanding the paper's limitations: the authors acknowledge that iterative pruning is computationally expensive (requiring 15+ training cycles) and that finding more efficient discovery methods is future work. The paper is demonstrating existence, not providing a practical recipe.

Finally, the paper frames the lottery ticket hypothesis as the first step toward several broader goals: designing training schemes that find winning tickets early, designing architectures and initializations inspired by winning ticket properties, and understanding the theoretical relationship between overparameterization and optimization. The hypothesis itself is not the destination—it's a lens through which to study these deeper questions.

3. Technical Approach

3.1 Reader Orientation

This section explains the experimental methodology for empirically validating the lottery ticket hypothesis. The "system" is not a deployed piece of software but rather a carefully controlled experimental protocol consisting of a network training regimen, a pruning procedure, and a comparison between different initialization conditions. The core problem the protocol solves is: given only a dense, randomly-initialized neural network and standard training data, how can we systematically identify whether a subnetwork capable of independent training exists within it, and how can we isolate whether its success comes from its architecture (which weights are present) or its initialization (what values those weights happen to have)?

3.2 Big-Picture Architecture (Diagram in Words)

The experimental system has five major components connected in a sequential pipeline:

1. Network Initialization Module — Randomly samples initial weights $\theta_0$ from a chosen distribution (Gaussian Glorot by default) and creates the dense network $f(x; \theta_0)$.

2. Training Module — Trains the network using a specified optimizer (Adam, SGD, or SGD with momentum), learning rate, batch size, and iteration budget on a dataset (MNIST or CIFAR10), producing trained weights $\theta_j$.

3. Pruning Module — Removes a percentage $p\%$ of the weights with the smallest magnitudes in each layer (or globally for deeper networks), creating a binary mask $m$ that zeroes out pruned weights.

4. Reset Module — Overwrites the surviving weights with their original values from $\theta_0$, producing the candidate winning ticket $f(x; m \odot \theta_0)$. This is the crucial step that distinguishes the lottery ticket hypothesis from standard pruning practice.

5. Evaluation and Comparison Module — Trains the candidate winning ticket in isolation, measures its early-stopping iteration and test accuracy, and compares against both the unpruned network and a version of the same sparse architecture that has been randomly reinitialized (to isolate initialization effects).

Information flows unidirectionally: initialization → training → pruning → reset → retraining (as a winning ticket) or reinitialized retraining → performance measurement → iteration (repeat steps 2–4 for iterative pruning, feeding the surviving weights and their original initializations back into training).

3.3 Roadmap for the Deep Dive

• First, the formal statement of the lottery ticket hypothesis, including its precise mathematical definition of what constitutes a winning ticket—this establishes the success criteria against which all experiments are judged.
• Second, the iterative pruning algorithm, which is the core mechanism for discovering winning tickets—this is the "search procedure" and understanding its mechanics is essential before seeing any results.
• Third, the random reinitialization experiment, which is the key control that isolates whether a winning ticket's success is due to its architecture or its initialization.
• Fourth, the one-shot pruning variant, which explores whether repeated training is necessary for finding winning tickets—this illuminates the cost/benefit of the iterative approach.
• Fifth, the extension to convolutional networks, introducing the Conv-2/4/6 architectures and addressing how pruning rates interact with different layer types.
• Sixth, the deep network variants (VGG-19 and Resnet-18), where learning rate sensitivity and global pruning become critical, along with the introduction of learning rate warmup.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical discovery paper whose core idea is that dense, randomly-initialized feed-forward networks contain sparse subnetworks whose original initialization enables them to be trained in isolation to match the full network's accuracy.

---

The Lottery Ticket Hypothesis — Formal Definition

The paper opens its technical exposition with a precise mathematical statement of what it means for a subnetwork to be a "winning ticket." This definition is essential because every subsequent experiment is designed to test whether such subnetworks exist.

Consider a dense feed-forward neural network $f(x; \theta)$ with initial parameters $\theta = \theta_0 \sim \mathcal{D}_\theta$, where $\mathcal{D}_\theta$ is the distribution from which weights are sampled (by default, Gaussian Glorot initialization, which sets the standard deviation of each layer's weights to $\sqrt{2 / (n_{\text{in}} + n_{\text{out}})}$ where $n_{\text{in}}$ and $n_{\text{out}}$ are the number of input and output units for that layer). When this network is trained using stochastic gradient descent (SGD) on a training set, it reaches minimum validation loss $\ell$ at iteration $j$ with test accuracy $a$.

Now, consider training the same network but with a binary mask $m \in \{0,1\}^{|\theta|}$ applied to its parameters, so that the network is $f(x; m \odot \theta)$, and its initialization is $m \odot \theta_0$ (the original initial weights of only the unmasked connections). When this masked network is optimized with SGD on the same training set with the mask $m$ held fixed, suppose it reaches minimum validation loss $\ell'$ at iteration $j'$ with test accuracy $a'$.

The lottery ticket hypothesis predicts that there exists a mask $m$ for which:

• $j' \leq j$ — the subnetwork trains in at most the same number of iterations (commensurate training time);
• $a' \geq a$ — the subnetwork achieves at least the same test accuracy (commensurate accuracy);
• $\|m\|_0 \ll |\theta|$ — the subnetwork uses far fewer parameters.

The notation $\|m\|_0$ denotes the $L_0$ "norm" (the number of non-zero entries in the mask), and $P_m = \|m\|_0 / |\theta|$ is the fraction of original weights that survive pruning. A winning ticket is then defined as a subnetwork $f(x; m \odot \theta_0)$ that satisfies all three conditions.

What this definition establishes: a winning ticket is not merely a sparse network that achieves high accuracy—it must satisfy three simultaneous constraints: it must train no slower than the original, reach accuracy no lower than the original, and use substantially fewer parameters. A network that achieves high accuracy but takes much longer to train would not qualify. A network that trains quickly but loses accuracy would not qualify. The three-part definition makes the hypothesis falsifiable: a single experiment showing that no such mask exists for a given architecture and dataset would refute it for that setting.

Why this particular definition: the authors are interested in whether sparse networks can be trained from scratch as a replacement for dense training. A winning ticket that trains slower defeats the practical purpose (training efficiency). A winning ticket with worse accuracy defeats the performance purpose. The conjunction of speed + accuracy + sparsity captures the claim that these subnetworks are genuinely better to train, not just smaller. The choice of early-stopping iteration $j$ as the speed metric (rather than, say, wall-clock time) is deliberate: it captures how many gradient updates are needed, abstracting away from hardware-specific timing while directly measuring optimization efficiency.

---

Iterative Pruning — The Discovery Algorithm

The paper's method for identifying winning tickets is iterative magnitude-based pruning with resetting. This is not proposed as a practical training algorithm—the authors explicitly acknowledge its computational expense—but rather as a discovery mechanism whose output can be studied to validate the lottery ticket hypothesis.

The procedure, formally called "Iterative pruning with resetting" (Strategy 1 in Appendix B), operates over multiple rounds. In the first round:

1. Randomly initialize a neural network $f(x; m \odot \theta)$, where $\theta = \theta_0$ and $m = \mathbf{1}^{|\theta|}$ is a mask of all ones (i.e., no weights pruned yet).
2. Train the network for $j$ iterations, producing parameters $m \odot \theta_j$.
3. Prune $s\%$ of the parameters. Specifically, within each layer independently (for Lenet and Conv-2/4/6), identify the connections whose absolute value $|\theta_j^{(i)}|$ is smallest, and remove the $s\%$ with the lowest magnitudes. This updates the mask $m$ to $m'$, where $P_{m'} = (P_m - s)\%$ of the weights remain.
4. Reset the surviving weights to their original values from $\theta_0$. That is, set $\theta = \theta_0$, so that the updated network is $f(x; m' \odot \theta_0)$.
5. Set $m = m'$ and repeat from step 2 until the network has been pruned to a sufficiently small size.

The key hyperparameters that control this process are:

• Pruning rate $s$: the percentage of surviving weights pruned in each round. For Lenet on MNIST, $s = 20\%$ per iteration for hidden layers and $s = 10\%$ per iteration for the output layer (the output layer is pruned at half the rate because it is so small—only 1,000 weights out of 266,000 total—that pruning it at full rate would eliminate it prematurely). For Conv-2, $s = 10\%$ for convolutional layers and $s = 20\%$ for fully-connected layers. For Conv-4, the same rates apply. For Conv-6, $s = 15\%$ for convolutional layers and $s = 20\%$ for fully-connected layers.

• Number of rounds $n$: determined implicitly by the pruning rate and the desired final sparsity. For example, pruning at $20\%$ per round from $100\%$ down to $0.3\%$ of weights remaining requires approximately $\log(P_{\text{final}}) / \log(1 - s) \approx 15$ rounds.

• Training iterations per round $j$: 50,000 for Lenet on MNIST; 20,000 for Conv-2, 25,000 for Conv-4, and 30,000 for Conv-6 on CIFAR10; 30,000 for Resnet-18; 112,480 for VGG-19 (160 epochs with batch size 64).

• Optimizer and learning rate: Lenet uses Adam with learning rate $1.2 \times 10^{-3}$. Conv-2 uses Adam with learning rate $2 \times 10^{-4}$. Conv-4 and Conv-6 use Adam with learning rate $3 \times 10^{-4}$. Resnet-18 and VGG-19 use SGD with momentum (0.9) and a learning rate schedule: starting at 0.1 (or 0.01 for some experiments), decreasing by a factor of 10 at specified iterations.

• Layer-wise vs. global pruning: For Lenet and Conv-2/4/6, pruning operates independently within each layer—each layer loses $s\%$ of its own weights. For VGG-19 and Resnet-18, pruning is global: the $s\%$ of weights with the smallest magnitudes are removed across all convolutional layers collectively, without regard to which layer they belong to. This prevents layers with fewer parameters from becoming bottlenecks that limit how far the network can be pruned (a layer with 1,728 parameters pruned at the same rate as a layer with 2.35 million parameters would be eliminated much sooner).

The reset step in (4) is what fundamentally distinguishes this procedure from standard pruning-and-fine-tuning. Standard practice is to retain the trained values of the surviving weights (Han et al., 2015) or to continue training the pruned network without resetting (Han et al., 2017). The authors' "Strategy 2" in Appendix B—iterative pruning with continued training, where the weights are NOT reset between rounds—is compared against Strategy 1, and Strategy 1 (with resetting) maintains higher validation accuracy and faster early-stopping times to smaller network sizes (Figures 9 and 10). The resetting step is what tests the lottery ticket hypothesis: if the surviving weights' original values are what matters, resetting should produce a trainable network; if only their trained values matter, resetting should destroy performance.

The authors also investigate a key alternative: pruning at iteration 0. That is, taking the initial weight magnitudes as the basis for pruning before any training has occurred. Appendix F.4 (Figure 18) shows that this performs even worse than random reinitialization of winning tickets, demonstrating that the information about which weights are important can only be extracted by the training process—the initial magnitudes alone are not predictive enough.

---

Isolating Initialization: The Random Reinitialization Experiment

To establish that a winning ticket's success is due to its initialization and not merely its structure, the paper employs a critical control experiment: random reinitialization.

After identifying a winning ticket's architecture $m$ via iterative pruning, the authors create a new network $f(x; m \odot \theta'_0)$ where $\theta'_0 \sim \mathcal{D}_\theta$ is freshly sampled from the same initialization distribution (e.g., Gaussian Glorot). This network has the exact same sparse connectivity pattern as the winning ticket but with different random weight values. It is then trained in isolation under identical conditions.

The comparison between the winning ticket $f(x; m \odot \theta_0)$ and the reinitialized version $f(x; m \odot \theta'_0)$ isolates the contribution of the specific weight values: if both networks train equally well, then architecture alone explains the success; if the winning ticket outperforms the reinitialized version, then the particular values of $\theta_0$ are essential.

What this experiment computes: the difference in learning speed and test accuracy between a sparse network whose weights are the original initializations that survived pruning, and an identically-structured sparse network whose weights are randomly sampled from the same distribution. Any gap measures the "initialization lottery" effect—how much better the specific initial weights are compared to random weights from the same distribution on the same architecture.

Why this form: this is the key falsification test for the lottery ticket hypothesis. If the structure alone (the mask $m$) were responsible for trainability—for example, if the pruning process simply identified a well-shaped sub-architecture that constrains the learning problem—then any random initialization of that architecture should perform similarly. The hypothesis specifically claims that the initialization itself matters, and the random reinitialization experiment directly tests this claim. Without this control, one could not distinguish between "the winning ticket has a good architecture" and "the winning ticket has fortuitous initial weights."

The experiment is conducted by reinitializing each winning ticket three times (for a total of 15 data points per mask, since each of the five winning ticket trials is reinitialized three times). This repetition is necessary because the reinitialized network's performance is stochastic (depending on the resampled weights), and averaging over multiple reinitializations reduces variance.

An even stronger version of this experiment appears in Appendix F.3: reinitializing from the empirical distribution of winning ticket initializations $D_m$. That is, rather than sampling from the original Gaussian, sampling new values by drawing from the actual histogram of surviving winning ticket weights—preserving not just the mean and variance but the exact shape (e.g., the bimodal distributions observed in Figure 15). This experiment tests whether the winning ticket's success comes from having a distinctive distribution of initializations (e.g., higher-magnitude weights or a particular bimodality) rather than the specific values themselves. The results (Figure 17) show that even this distribution-matched reinitialization performs far worse than the original winning ticket, indicating that the exact values—not just their statistical properties—are what matter.

---

One-Shot Pruning: Trading Discovery Cost for Ticket Quality

Iterative pruning requires training the network 15+ times consecutively, which is computationally expensive. One-shot pruning attempts to find winning tickets with a single training run. The procedure is:

1. Randomly initialize the network $f(x; \theta_0)$.
2. Train for $j$ iterations, producing $\theta_j$.
3. Prune $p\%$ of the parameters in one shot (rather than $s\%$ per round across multiple rounds), creating mask $m$.
4. Reset surviving weights to $\theta_0$, producing $f(x; m \odot \theta_0)$.

This tests whether the iterative refinement of the mask is necessary, or whether a single round of training provides enough information to identify a winning ticket.

The comparison between one-shot and iterative pruning (Figure 4, green vs. blue lines) reveals that one-shot pruning does find winning tickets—when $67.5\% \geq P_m \geq 17.6\%$, the average winning tickets reach minimum validation accuracy earlier than the original network, and when $95.0\% \geq P_m \geq 5.17\%$, test accuracy is higher. However, iteratively-pruned winning tickets learn faster and maintain high accuracy at smaller network sizes. Specifically, the iterative winning ticket at $P_m = 3.6\%$ matches the original network's early-stopping iteration and accuracy, while one-shot pruning fails to produce viable winning tickets below about $5\%$.

What this tells us about the pruning process: the iterative approach does not merely provide a finer-grained pruning schedule—it fundamentally changes which weights are identified as important. Between rounds, the network is reset and retrained, meaning that in each round, the pruning decision is based on the retrained weights of the surviving connections starting from their original initializations. This allows the training process to "re-explore" the weight space at each round, potentially identifying different connections as low-magnitude than in the previous round. One-shot pruning, by contrast, makes all pruning decisions based on a single trajectory through weight space, which may lock in early decisions that prove suboptimal at higher sparsity levels.

Why this comparison matters: if one-shot pruning were sufficient, the lottery ticket hypothesis could be validated with dramatically less computation, and practical training schemes based on winning ticket discovery would be far more feasible. The fact that iterative pruning finds smaller winning tickets suggests that the discovery process benefits from the repeated exploration that resetting enables—which in turn hints that the lottery ticket phenomenon is genuinely about the interaction between initialization and training dynamics, not just about identifying which weights end up small after training.

---

Extension to Convolutional Networks

The paper extends the lottery ticket experiments from the fully-connected Lenet architecture on MNIST to convolutional architectures on CIFAR10, testing whether winning tickets exist in networks with substantially different structure. The Conv-2, Conv-4, and Conv-6 architectures (Figure 2) are scaled-down variants of VGG (Simonyan & Zisserman, 2014), designed to span a range from near-fully-connected (Conv-2, where convolutional layers hold less than 1% of total parameters) to traditionally convolutional (Conv-6, where convolutions hold nearly two-thirds of parameters).

Each architecture consists of a series of modules, where each module has two 3×3 convolutional layers followed by 2×2 max-pooling with stride 2. Conv-2 has one module (64, 64 filters), Conv-4 has two modules (64, 64 in the first, 128, 128 in the second), and Conv-6 has three modules (64, 64; 128, 128; 256, 256). All are followed by two fully-connected layers of 256 units and a 10-unit output layer. The total parameter counts are 4.3M (Conv-2, with 38K in convolutions), 2.4M (Conv-4, with 260K in convolutions), and 1.7M (Conv-6, with 1.1M in convolutions).

The training regimes differ from Lenet in several important ways:

• Iteration budgets: Conv-2 is trained for 20,000 iterations, Conv-4 for 25,000, and Conv-6 for 30,000—substantially fewer than Lenet's 50,000 because CIFAR10 has 50,000 training images compared to MNIST's 60,000, and the convolutional networks have more parameters, making each iteration more informative.
• Batch size: 60 for all three architectures.
• Optimizer: Adam with learning rates $2 \times 10^{-4}$ (Conv-2), $3 \times 10^{-4}$ (Conv-4), and $3 \times 10^{-4}$ (Conv-6). These were selected from sweeps (Appendix H.2, Figure 32) using the criteria of maximizing initial accuracy, maintaining high accuracy and fast early-stopping to the smallest possible network sizes, and being conservative enough to work across a range of hyperparameters.
• Pruning rates: convolutional layers and fully-connected layers are pruned at different rates, with the rates selected per-network (Appendix H.4, Figure 35). Conv-2 prunes convolutional layers at 10% and fully-connected layers at 20% per round. Conv-4 uses the same rates. Conv-6 prunes convolutional layers at 15% and fully-connected layers at 20% per round. These differential rates reflect the relative importance of each layer type—in Conv-2 and Conv-4, fully-connected layers dominate the parameter count and pruning them too aggressively would bottleneck before convolutions could be meaningfully reduced; in Conv-6, convolutional layers comprise the majority of parameters and need a higher pruning rate to avoid being the bottleneck.
• Output layer pruning: The output layer is pruned at half the rate of the fully-connected layers (10% when FC is 20%), as with Lenet, because the output layer is very small and would be eliminated early otherwise.

The key architectural insight explored in a dedicated ablation (Appendix H.6, Figure 37) is the effect of pruning convolutions alone versus fully-connected layers alone versus both. Across all three architectures, pruning convolutions alone leads to higher test accuracy and faster learning, while pruning fully-connected layers alone causes accuracy to worsen and learning to slow. However, pruning convolutions alone has limited ability to reduce the overall parameter count (only 1%, 11%, and 65% of parameters for Conv-2, Conv-4, and Conv-6 respectively), so both must be pruned to achieve substantial compression.

---

Dropout Interaction

Given that dropout (Srivastava et al., 2014) randomly disables a fraction of units on each training iteration—effectively training a random subnetwork each step—the paper asks whether dropout interacts with winning ticket discovery. Baldi & Sadowski (2013) characterize dropout as simultaneously training the ensemble of all subnetworks, and since the lottery ticket hypothesis suggests one of these subnetworks is a winning ticket, there is reason to expect a connection.

The experiments (Section 3, Figure 6) add dropout with rate 0.5 to Conv-2, Conv-4, and Conv-6 during training. Because dropout slows learning, training iterations are tripled: 60,000 for Conv-2, 75,000 for Conv-4, and 90,000 for Conv-6. Learning rates are adjusted accordingly: $3 \times 10^{-4}$ for Conv-2, and $2 \times 10^{-4}$ for Conv-4 and Conv-6 (selected via sweeps in Appendix H.5, Figure 36).

The results show that dropout increases initial test accuracy: Conv-2 improves by 2.1 percentage points, Conv-4 by 3.0, and Conv-6 by 2.4. Iterative pruning further improves accuracy on top of dropout (up to 2.3, 4.6, and 4.7 additional percentage points, respectively). Learning becomes faster with iterative pruning, but less dramatically than without dropout for Conv-2.

Why this is notable: the fact that winning tickets are still found when dropout is active suggests that the lottery ticket phenomenon is robust to the stochastic regularization that dropout introduces. The complementary nature of the accuracy improvements—dropout helps, and iterative pruning helps more on top of dropout—suggests that dropout and pruning operate through different mechanisms. Srivastava et al. (2014) observed that dropout induces sparse activations in the final network; the authors hypothesize that "dropout-induced sparsity primes a network to be pruned," perhaps by making the network less reliant on any individual connection, which in turn makes magnitude-based pruning more effective at identifying redundancies.

---

Deep Networks: VGG-19 and Resnet-18

The paper extends the lottery ticket experiments to two architectures representative of practical deep learning: VGG-19 (Simonyan & Zisserman, 2014) adapted for CIFAR10 (following Liu et al., 2019) and Resnet-18 (He et al., 2016) for CIFAR10. These architectures introduce complications absent from the shallower networks: batch normalization, weight decay, learning rate schedules, data augmentation, and (for Resnet-18) residual connections.

VGG-19 comprises five groups of 3×3 convolutional layers (two layers in the first two groups, four in the last three), with max-pooling (stride 2) after each of the first four groups and average pooling after the last, followed by a single fully-connected output layer. The total parameter count is 20 million, making it nearly two orders of magnitude larger than Conv-6 (1.7M). Training follows Liu et al. (2019): 160 epochs (112,480 iterations) with SGD with momentum (0.9), batch size 64, weight decay 0.0001, data augmentation, and a learning rate schedule where the rate decreases by a factor of 10 at 80 and 120 epochs.

Resnet-18 comprises a convolutional layer followed by nine pairs of 3×3 convolutional layers (with residual connections around each pair), average pooling, and a fully-connected output layer. It has 271,000 parameters—comparable to Conv-4 in scale but structurally very different due to the residual connections. Training follows He et al. (2016): 30,000 iterations with SGD with momentum (0.9), batch size 128, weight decay 0.0001, data augmentation (random flips and four-pixel pads and crops), and learning rate drops by 10 at 20,000 and 25,000 iterations.

Global pruning. For these deeper networks, the authors switch from layer-wise pruning to global pruning: the $s\%$ of weights with the smallest magnitudes are removed collectively across all convolutional layers, without regard to which layer they come from. Layer-wise pruning identifies smaller winning tickets for VGG-19 and Resnet-18 because some layers have vastly more parameters than others—in VGG-19, the first convolutional layer has 1,728 parameters while the last has 2.35 million. Pruning all layers at the same rate would eliminate the small layers much sooner than the large ones, creating a capacity bottleneck before the network as a whole is significantly compressed. Global pruning allows the pruning procedure to allocate the parameter budget where it is most needed.

For VGG-19, convolutional layers are pruned at 20% per iteration globally; the 5,120 parameters in the output layer are not pruned. For Resnet-18, convolutions are pruned at 20% per iteration globally; the 2,560 parameters in residual downsampling connections and the 640 parameters in the output layer are not pruned due to their negligible fraction of the total.

The learning rate sensitivity challenge. The most surprising finding for these deeper networks is that iterative pruning is highly sensitive to the learning rate. At the standard learning rate of 0.1 (used in both Liu et al. for VGG-19 and He et al. for Resnet-18), iterative pruning does not find winning tickets. Instead, the pruned networks with their original initialization perform no better than when randomly reinitialized—both degrade similarly as sparsity increases (Figures 7 and 8, left panels for rate 0.1).

However, at a lower learning rate of 0.01, the lottery ticket pattern reemerges: the winning tickets maintain accuracy to much smaller sizes than randomly reinitialized networks. For VGG-19 at rate 0.01, winning tickets remain within 1 percentage point of the original accuracy when $P_m \geq 3.5\%$, whereas reinitialized networks lose accuracy much more rapidly. For Resnet-18 at rate 0.01, winning tickets reach up to 89.5% accuracy (when $41.7\% \geq P_m \geq 21.9\%$), though this still falls short of the 90.5% achieved at rate 0.1 by the unpruned network.

The explanation for this sensitivity is not fully resolved, but the authors' characterization is that at higher learning rates, the iterative pruning and resetting procedure disrupts the optimization dynamics in a way that prevents the network from leveraging its initialization. At the higher learning rate, the gradients are larger and the optimization takes more aggressive steps; when those steps are taken from the reset initial weights in each round, the pruned network may diverge from the trajectory that would have identified productive weight configurations.

Learning rate warmup as a solution. To bridge the gap between the lottery ticket behavior at lower learning rates and the superior accuracy at higher learning rates, the authors introduce linear learning rate warmup: the learning rate starts at 0 and increases linearly to its target value over the first $k$ iterations. The intuition is that warmup gives the randomly initialized weights time to settle into a productive configuration before large gradient steps are taken, making the training process more robust to the pruning and resetting cycle.

For VGG-19, warmup with $k = 10{,}000$ iterations at learning rate 0.1 improves the unpruned accuracy by about 1 percentage point, and makes it possible to find winning tickets that exceed this improved accuracy when $P_m \geq 1.5\%$—a dramatic improvement over the layer-wise pruning limit of $P_m \geq 6.9\%$ at the same warmup setting (Appendix I.1, Figure 38). The choice of $k = 10{,}000$ comes from a sweep (Appendix I.5, Figure 45) showing that accuracy improves rapidly for $k$ up to 5,000 and reaches diminishing returns beyond that.

For Resnet-18, warmup with $k = 20{,}000$ at learning rate 0.03 allows winning tickets to match the 90.5% accuracy of the 0.1-rate unpruned network (something the 0.01 rate alone could not achieve) while still finding winning tickets when $P_m \geq 11.8\%$. However, the authors note they could not find hyperparameters with which to identify winning tickets at the full 0.1 learning rate even with warmup—$0.03$ was the highest rate that worked, suggesting a fundamental limitation in the pruning-and-resetting procedure's compatibility with very aggressive optimization.

The need for warmup is a significant finding: it demonstrates that the lottery ticket phenomenon, while real, depends on the training dynamics being sufficiently "gentle" to preserve the information in the winning ticket's initialization. Aggressive optimization (high learning rates, large momentum) appears to overwrite or discard the initialization signal that makes winning tickets special, which is why warmup—by gradually ramping up the optimization intensity—restores the effect.

4. Key Insights and Innovations

Innovation 1: Reframing Pruning as a Search for a Pre-Existing Subnetwork, Not a Compression of a Learned Solution

The paper's most fundamental conceptual move is its reframing of what pruning does. Prior work treated pruning as a compression technique: train a large network to learn a function, then remove redundant parameters that don't contribute to that learned function, and fine-tune what remains. In this view, the initial training run serves two inseparable purposes — it discovers which connections are important AND it sets their values to a good configuration. The pruned architecture without those trained values is just a skeleton, not independently useful.

The lottery ticket hypothesis proposes a radically different interpretation: the dense network at initialization already contains a sparse subnetwork whose connections and initial values are independently sufficient for learning. Training the full network is not constructing this subnetwork but discovering it. The trained weights on surviving connections are incidental — what matters is that the connections that survive pruning had initial values that made them intrinsically more trainable than other connections. The goal of pruning, in this light, is not compression but excavation — exposing a latent structure that was present from the start.

This reframing matters because it shifts the research question entirely. Instead of asking "how can we compress trained networks without losing accuracy?" one asks "what properties of initialization make a subnetwork trainable, and can we identify those subnetworks without training the full network first?" The former is an engineering optimization problem; the latter is a scientific question about the nature of neural network initialization and optimization.

The key evidence supporting this reframing is the resetting step itself. If pruning were merely identifying a compressed version of the learned function, resetting surviving weights to their initial (pre-training) values would destroy performance. The fact that reset networks outperform the original dense network — training faster and generalizing better — is direct evidence that the pruned architecture was not dependent on the trained weight values. The trained values were a vehicle for discovery, not the payload.

This contrasts sharply with the contemporaneous framing of Liu et al. (2019), who argued that the value of pruning was primarily in discovering good architectures (which could then be randomly reinitialized), questioning whether the trained weights mattered at all. The lottery ticket hypothesis splits the difference: the architecture AND the initialization both matter, but the initialization that matters is the original one, not the trained one. Liu et al. showed that architecture alone could suffice at moderate sparsity; this paper shows that at extreme sparsity, the original initialization becomes indispensable — the trained weights never were.

Innovation 2: The Random Reinitialization Control as a Diagnostic for Distinguishing Architecture from Initialization

Prior pruning work lacked a clean experimental design for separating the contributions of a sparse network's structure (which connections are present) from the contributions of its weight values. Standard practice — fine-tune the surviving weights from their trained values — conflates the two: you cannot tell whether the pruned network works because of what it looks like or because of what its weights happen to be. Even when prior work reinitialized pruned networks, it typically did so as a negative result ("training from scratch fails"), not as a systematic diagnostic.

The paper's random reinitialization experiment is a conceptual innovation disguised as a control. By taking the exact mask $m$ discovered by pruning and resampling the weights from the same distribution, the experiment isolates the contribution of the specific weight values to performance. The comparison between $f(x; m \odot \theta_0)$ and $f(x; m \odot \theta'_0)$ measures the "initialization lottery" effect — how much better the original initial weights are than random weights from the same distribution on the same architecture.

This diagnostic yields a clean, falsifiable test of the lottery ticket hypothesis: if randomly reinitialized winning tickets perform as well as the originals, the hypothesis is false — structure alone explains success. If they perform worse, initialization matters. The experiments unambiguously show the latter: for Lenet on MNIST, a winning ticket at $P_m = 21.1\%$ reaches early-stopping 2.5× faster and is half a percentage point more accurate than its randomly reinitialized counterpart (Figure 4a, orange vs. blue). For Conv-2/4/6, randomly reinitialized networks learn increasingly slower upon continued pruning while winning tickets learn increasingly faster (Figure 5, dashed vs. solid lines).

The diagnostic is enriched by the additional controls in the appendices: reinitializing from the empirical distribution of winning ticket weights (Appendix F.3, Figure 17) and adding Gaussian noise to winning ticket initializations (Appendix F.7, Figure 25). The former tests whether the mere statistical properties of the winning ticket's initializations (e.g., bimodality, higher magnitude) are sufficient — they aren't. The latter shows that winning tickets are surprisingly robust to noise (adding noise of $0.5\sigma$ barely changes accuracy) but progressively degrade as more noise is added, eventually approaching random reinitialization performance at $3\sigma$. Together, these experiments paint a nuanced picture: the exact values matter, but they lie in a relatively broad basin — it's not that a single set of weights works; it's that the winning ticket initialization occupies a region of weight space where the architecture becomes trainable, and random initialization from the original distribution typically falls outside that region.

Innovation 3: Difficulty-Dependent Behavior of Winning Ticket Discovery — the Learning Rate Sensitivity and Warmup Requirement

The paper makes a non-obvious empirical discovery that fundamentally constrains when and how winning tickets can be found. On deeper networks (VGG-19 and Resnet-18), iterative pruning fails to find winning tickets at the standard high learning rates used to train these networks (0.1). Instead, the pruned networks with original initialization perform identically to randomly reinitialized ones — the initialization lottery appears not to exist (Figures 7 and 8, rate 0.1 panels). Yet at lower learning rates (0.01), the lottery ticket pattern reemerges cleanly.

This is not a minor hyperparameter sensitivity. It is a discovery about the relationship between optimization aggressiveness and the preservation of initialization information. At high learning rates, SGD takes large steps that rapidly move weights away from their initial values; when those weights are then reset and retrained in the next pruning round, the optimization dynamics appear to "forget" which initial configurations were productive. The initialization signal that makes winning tickets special is being overwritten by the training process faster than the pruning procedure can capture it.

The introduction of learning rate warmup as a solution is both practically important and theoretically revealing. Warmup — linearly increasing the learning rate from 0 to its target over the first $k$ iterations — allows winning tickets to be found at higher learning rates (0.03 for Resnet-18, 0.1 for VGG-19). The mechanism is not explained in detail, but the implication is clear: warmup gives the initial weights time to settle into productive configurations before large gradient steps are taken, preserving the relationship between initialization quality and eventual trained weight magnitude that the pruning procedure relies on. When the optimization is too aggressive from the start, this relationship is destroyed.

This finding has a significant practical implication: it means the lottery ticket hypothesis is not a universal property of all training regimes. Whether winning tickets can be discovered depends on the optimization algorithm, its hyperparameters, and the interaction between the pruning schedule and the learning dynamics. The authors' conjecture that "SGD seeks out and trains a well-initialized subnetwork" (Section 5) implies that at high learning rates without warmup, SGD may not be successfully doing so — it may be converging to solutions that do not correspond to any fortuitously-initialized subnetwork, or the correspondence between initialization and final weight magnitude (which magnitude-based pruning relies on) may break down.

The broader significance is that this finding opens a line of inquiry into why optimization hyperparameters interact with the lottery ticket phenomenon. If winning tickets exist at initialization but can only be found under certain training conditions, then understanding those conditions is essential both for practical winning ticket discovery and for theoretical understanding of why overparameterization helps optimization. It's not simply that larger networks contain more subnetworks — it's that SGD under typical (but not all) hyperparameters acts as an implicit search over these subnetworks, and the search can fail.

Innovation 4: Empirical Evidence for Overparameterization as a Subnetwork Search Space, Not a Smoothing Mechanism

The paper offers a novel mechanistic hypothesis for why overparameterized networks are easier to train — a question that was receiving substantial theoretical attention at the time. Du et al. (2019) had shown theoretically that sufficient overparameterization guarantees convergence to global optima for two-layer ReLU networks, framing the benefit in terms of the optimization landscape becoming more benign. The lottery ticket hypothesis suggests a fundamentally different mechanism: larger networks are easier to train because they contain a combinatorially larger number of possible subnetworks, increasing the probability that at least one of them is a "winning ticket" with a fortuitous initialization.

The evidence for this interpretation comes from the network size scaling experiments in Appendix G.6 (Figure 31). When networks of different initial sizes (from 25/9 to 900/300 hidden units, always in 3:1 ratio) are pruned to the same absolute number of weights, the winning tickets from initially larger networks achieve higher test accuracy. This means that the larger initial network does not just provide more parameters — it provides a richer space of possible subnetworks, and the ones selected by pruning are qualitatively better. The accuracy benefit of initial overparameterization persists even after pruning to identical parameter counts, suggesting the selection process has access to better candidates in the larger pool.

This is not a proof, and the authors are careful to present it as a conjecture. But it is a productive conjecture because it makes testable predictions that differ from the "optimization landscape smoothing" view. If overparameterization helps primarily by providing a larger subnetwork search space, then (a) winning tickets should be transferable (a subnetwork that works for one task might work for another, since its quality is intrinsic to its initialization, not its fit to a particular dataset); (b) the benefit of overparameterization should saturate once the network is large enough to contain a winning ticket with high probability; and (c) architectures that are easier to prune (i.e., where magnitude-based pruning more reliably finds the best subnetwork) should benefit less from overparameterization. The paper does not test these predictions, but they naturally follow from the framing and provide a roadmap for distinguishing between competing theories of overparameterization.

The connection to generalization is similarly novel. The paper observes an "Occam's Hill" (Rasmussen & Ghahramani, 2001) — test accuracy increases as the network is pruned, peaks at moderate sparsity, and then declines. The lottery ticket hypothesis explains this in terms of subnetwork selection rather than complexity regularization: the original overparameterized network contains the winning ticket (which generalizes well) alongside many other subnetworks (which may overfit); training the full network averages over all of them, while training the isolated winning ticket benefits from its specific inductive bias. The improved generalization of winning tickets (training accuracy reaches 100% for nearly all sparsity levels, but test accuracy is higher for winning tickets — Appendix D, Figures 12 and 13) is thus not a result of pruning acting as regularization, but of pruning selecting a subnetwork that was already better at generalization from the moment of initialization.

Innovation 5: Winning Tickets as a Unifying Lens on Previously Disconnected Phenomena

The paper's final conceptual contribution is demonstrating that several previously disconnected observations about neural network training can be understood through the single lens of the lottery ticket hypothesis. This is not a claim the paper makes explicitly in a single section, but it emerges from the way the paper structures its investigation.

The connection between pruning and trainability. Prior work had treated the difficulty of training pruned architectures from scratch as a practical nuisance — Li et al. (2016) and Han et al. (2015) simply noted that reinitialization fails and moved on. The lottery ticket hypothesis explains why: the architecture alone is insufficient; the initialization is essential, and standard pruning preserves it implicitly because it retains trained weights. The conventional wisdom that "pruned networks are hard to train from scratch" is revealed to be a statement about initialization, not about sparse architectures being inherently untrainable.

The connection between dropout and pruning. Srivastava et al. (2014) observed that dropout induces sparse activations, and Baldi & Sadowski (2013) characterized dropout as training an ensemble of subnetworks. The lottery ticket hypothesis connects this to pruning: if dropout is sampling subnetworks during training, it may be providing a training signal that makes magnitude-based pruning more effective at identifying winning tickets — dropout-induced sparsity "primes" the network by forcing it to function under many different connectivity patterns, making individual connections less critical and making the set of genuinely important connections (the eventual winning ticket) more clearly identifiable via their trained magnitudes.

The connection between compression and generalization. Theoretical results by Zhou et al. (2018) and Arora et al. (2018) had shown that networks that can be compressed further have tighter generalization bounds. The lottery ticket hypothesis offers a mechanism: compression (pruning) reveals the subnetwork that was actually doing the learning, and that subnetwork's generalization properties were inherent to its initialization. The relationship between compressibility and generalization is thus not causal (compression doesn't cause better generalization); rather, both are consequences of the same underlying structure — the presence of a well-initialized, well-generalizing subnetwork within the larger dense network.

This unifying power is what elevates the lottery ticket hypothesis from an empirical observation to a scientific contribution. It doesn't just describe a new phenomenon; it reinterprets existing phenomena through a consistent framework, resolving apparent contradictions (e.g., Liu et al. finding that reinitialization works at moderate sparsity while this paper shows it fails at extreme sparsity — both are true, and the lottery ticket hypothesis explains the boundary between them) and suggesting new connections between previously separate research areas.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on two standard vision benchmarks: MNIST (LeCun et al., 1998) for fully-connected network experiments and CIFAR10 (Krizhevsky & Hinton, 2009) for convolutional network experiments. MNIST contains 60,000 training and 10,000 test grayscale images of handwritten digits (28×28 pixels, 10 classes). CIFAR10 contains 50,000 training and 10,000 test color images (32×32 pixels, 10 classes). For both datasets, the authors randomly sample a 5,000-example validation set from the training split, leaving 55,000 (MNIST) or 45,000 (CIFAR10) training examples. Validation data is used only for determining early-stopping iteration retroactively; it never influences training. Test data is used for final evaluation.

• Base model(s). The paper studies six architectures spanning a range of depths and structural types, all summarized in Figure 2:

  • Lenet (LeCun et al., 1998): a fully-connected network with two hidden layers of 300 and 100 units, plus a 10-unit output layer. Total parameters: 266K. Operates on MNIST.
  • Conv-2: a shallow convolutional network with one VGG-style module (two 3×3 convolutional layers with 64 filters each, followed by 2×2 max-pooling), then two fully-connected layers of 256 units and a 10-unit output. Total parameters: 4.3M (only 38K in convolutions — <1%). Operates on CIFAR10.
  • Conv-4: two modules (64, 64; 128, 128 filters), same FC structure. Total: 2.4M parameters (260K in convolutions — ~11%).
  • Conv-6: three modules (64, 64; 128, 128; 256, 256 filters), same FC structure. Total: 1.7M parameters (1.1M in convolutions — ~65%).
  • VGG-19 (Simonyan & Zisserman, 2014, adapted for CIFAR10 by Liu et al., 2019): a deep convolutional network with five groups of 3×3 convolutional layers (2, 2, 4, 4, 4 layers respectively), max-pooling after each of the first four groups, average pooling after the last, and a single fully-connected output layer. Total: 20M parameters. Trained with batch normalization, weight decay (0.0001), data augmentation, and a staged learning rate schedule.
  • Resnet-18 (He et al., 2016, adapted for CIFAR10): a 20-layer residual network with nine pairs of 3×3 convolutional layers with skip connections, average pooling, and a fully-connected output. Total: 271K parameters. Trained with batch normalization, weight decay (0.0001), data augmentation, and a staged learning rate schedule.

  The architectural diversity is deliberate: Lenet tests the hypothesis on purely fully-connected networks; Conv-2/4/6 span a spectrum from near-fully-connected to traditionally convolutional; VGG-19 and Resnet-18 test whether winning tickets exist in deeper, practically-relevant architectures with modern training techniques. All networks use Gaussian Glorot initialization (Glorot & Bengio, 2010) by default.

• Metrics. Three quantities are measured throughout:

  • Early-stopping iteration: the training iteration at which validation loss reaches its minimum. This serves as a proxy for "how quickly the network learns" — networks that reach minimum validation loss sooner are considered to learn faster. The paper measures this by evaluating validation loss every 100 iterations and retroactively identifying the minimum. This metric is not used for VGG-19 and Resnet-18 due to their staged learning rate schedules (which make the early-stopping point less interpretable); for these networks, accuracy at several fixed iteration counts is reported instead.
  • Test accuracy at early-stopping: the accuracy on the held-out test set at the iteration when validation loss was minimized. This captures whether faster learning comes at the cost of final performance.
  • Test accuracy at the final training iteration: reported for experiments where training accuracy reaches ~100% (Appendix D, Figures 12 and 13), to assess whether winning tickets generalize better even after full convergence on the training set.

  All metrics are averaged over multiple independent trials (typically five for main experiments, three for hyperparameter sweeps in appendices), with error bars showing minimum and maximum values.

• Baselines. The paper compares winning tickets against three conditions:

  • Original unpruned network: the dense network $f(x; \theta_0)$ trained from scratch. This is the reference point against which winning tickets must match or exceed both accuracy and training speed.
  • Randomly reinitialized winning ticket: the same sparse architecture as the winning ticket (same mask $m$), but with freshly sampled weights $\theta'_0 \sim \mathcal{D}_\theta$. This is the key control that isolates whether the winning ticket's success comes from its architecture or its specific initialization.
  • Random sparse subnetworks: for several experiments (Appendix E, Figure 14), networks with the same number of parameters as winning tickets but with randomly chosen connections (i.e., random masks). This tests whether any sparse network of that size would perform similarly, or whether the specific connections identified by pruning are special.

  Additionally, the paper compares two pruning strategies (Appendix B, Figures 9 and 10):

  • Strategy 1 (resetting): after each pruning round, surviving weights are reset to their original initialization before retraining. This is the strategy used in all main experiments.
  • Strategy 2 (continued training): after each pruning round, training continues from the current weight values without resetting. This is closer to standard pruning practice and serves as an ablation of the resetting step.

• Generation budget / compute accounting. The paper does not measure FLOPs directly. Instead, the primary unit of computation is training iterations (number of gradient updates). Fair comparison is enforced by training all networks — original, winning ticket, and reinitialized — for the same number of iterations under the same optimizer and batch size. The "cost" of discovering winning tickets via iterative pruning (training the network 15+ times consecutively) is acknowledged as a limitation but is not the focus of the analysis; the experiments aim to demonstrate existence, not to propose a computationally efficient discovery method. The computational expense is partly why the paper only studies MNIST and CIFAR10, not larger datasets like ImageNet — the authors note that "iterative pruning is computationally intensive, requiring training a network 15 or more times consecutively for multiple trials" (Section 6).

• Cross-validation / statistical protocol. The paper does not use cross-validation in the traditional sense — there is no hyperparameter tuning on held-out folds of the test set. Instead:

  • A fixed validation set (5,000 examples randomly sampled from the training set) is used for all hyperparameter selection (learning rates, pruning rates, warmup duration) and for determining early-stopping points retroactively.
  • The test set is used exactly once for final evaluation and is never consulted during training or hyperparameter selection.
  • Each experiment is repeated multiple times (five trials for main experiments, three for most appendix sweeps), and results are reported as averages with min/max error bars. This captures variance due to random initialization and stochastic training.
  • For random reinitialization experiments, each of the five winning ticket masks is reinitialized three times, producing 15 data points per condition. This is necessary because reinitialized performance is stochastic and averaging reduces variance.
  • For hyperparameter sweeps in Appendices G, H, and I, the selection criteria are explicitly stated: (1) maximize validation accuracy and minimize early-stopping iteration on the unpruned network, (2) allow winning tickets to maintain these properties to the smallest possible network size, and (3) prefer conservative (slower) hyperparameters likely to work across varied conditions.

---

Main Quantitative Results

Winning Tickets in Fully-Connected Networks (Lenet on MNIST)

Iterative pruning results (Section 2, Figures 3 and 4a). The Lenet architecture trained with Adam (learning rate $1.2 \times 10^{-3}$) on MNIST for 50,000 iterations exhibits the full lottery ticket pattern when iteratively pruned at 20% per round (10% for the output layer). The headline numbers, reported as averages of five trials:

• Faster learning: As $P_m$ (the percentage of weights remaining) decreases from 100% to 21.1%, the early-stopping iteration drops substantially — at $P_m = 21.1\%$, early-stopping occurs 38% earlier than for the original unpruned network (Figure 4a, left panel, blue line). Further pruning slows learning again: at $P_m = 3.6\%$, the winning ticket's early-stopping iteration returns to approximately the same level as the original network.

• Higher test accuracy: Test accuracy at early-stopping increases with pruning, peaking at $P_m = 13.5\%$ with an improvement of more than 0.3 percentage points over the original network (Figure 4a, middle panel; the original network achieves approximately 97.8–97.9% test accuracy, and the winning ticket at $P_m = 13.5\%$ achieves approximately 98.2%). As pruning continues, accuracy declines, returning to the original level when $P_m = 3.6\%$.

• Training accuracy at early-stopping also increases (Figure 4a, right panel), rising with pruning in a pattern similar to test accuracy. This could initially suggest that winning tickets merely optimize faster without generalizing better. However, when examined at iteration 50,000 (Figure 4b), training accuracy reaches ~100% for $P_m \geq 2\%$ across all conditions (Appendix D, Figure 12 confirms this), yet winning tickets still maintain a test accuracy advantage of up to 0.35 percentage points over the original network. This means the gap between training and test accuracy is smaller for winning tickets — they generalize better even after fully fitting the training data.

• Size of winning tickets: The iterative pruning procedure finds winning tickets that match or exceed the original network's performance down to $P_m = 3.6\%$ of the original weights (approximately 9,600 parameters out of 266,000). Below this, test accuracy drops below the original level. The smallest winning ticket at $P_m = 3.6\%$ trains no slower and no less accurately than the original dense network.

Random reinitialization (Section 2, Figure 4a, orange line). When the same winning ticket architectures are randomly reinitialized (three times per mask, 15 data points per $P_m$), the results are starkly different:

• Learning becomes increasingly slower with pruning rather than faster. At $P_m = 21.1\%$, the winning ticket reaches minimum validation loss 2.51× faster than its randomly reinitialized counterpart.
• Test accuracy drops off much sooner: reinitialized networks lose accuracy relative to the original when $P_m$ drops below 21.1%, compared to 2.9% for the winning ticket. The winning ticket at $P_m = 21.1\%$ is half a percentage point more accurate than the reinitialized version.
• This divergence widens as pruning continues, demonstrating that the winning ticket's success is due to its specific initialization, not just its sparse architecture.

One-shot vs. iterative pruning (Section 2, Figure 4c and Figure 4a). One-shot pruning (training once, pruning $p\%$ of weights, resetting) does find winning tickets: when $67.5\% \geq P_m \geq 17.6\%$, one-shot winning tickets reach minimum validation loss earlier than the original network; when $95.0\% \geq P_m \geq 5.17\%$, test accuracy exceeds the original. However, the winning tickets found by one-shot pruning are larger (cannot be pruned as far) and perform worse at the same sparsity levels than iteratively-pruned tickets. On the logarithmic axes of Figure 4a, the green (one-shot) line sits above the blue (iterative) line for early-stopping iteration and below it for test accuracy — iterative pruning consistently finds smaller, faster, more accurate winning tickets. The smallest viable one-shot winning ticket ($P_m \approx 5\%$) is larger than the smallest iterative winning ticket ($P_m \approx 3.6\%$), and at comparable sparsity levels, iterative tickets learn faster (earlier early-stopping) and achieve higher accuracy.

---

Winning Tickets in Convolutional Networks (Conv-2/4/6 on CIFAR10)

Iterative pruning results (Section 3, Figure 5, solid lines). The Conv-2, Conv-4, and Conv-6 architectures trained with Adam on CIFAR10 all exhibit the lottery ticket pattern, with results even more pronounced than for Lenet:

• Conv-2 ($P_m \geq 2\%$ viable winning tickets):

  • Early-stopping iteration improves by up to 3.5× at $P_m = 8.8\%$ (Figure 5, top left, blue line).
  • Test accuracy at early-stopping improves by up to 3.4 percentage points at $P_m = 4.6\%$ (from approximately 67.6% for the original network to roughly 71.0%).
  • At iteration 20,000 (end of training), test accuracy remains elevated for winning tickets even though training accuracy reaches 100% for $P_m \geq 2\%$ (Appendix D, Figure 13), confirming improved generalization.

• Conv-4 ($P_m \geq 2\%$ viable winning tickets):

  • Early-stopping iteration improves by up to 3.5× at $P_m = 9.2\%$ (Figure 5, middle left, orange line).
  • Test accuracy improves by up to 3.5 percentage points at $P_m = 11.1\%$ (from approximately 74.9% to roughly 78.4%).

• Conv-6 ($P_m \geq 2\%$ viable winning tickets):

  • Early-stopping improves by up to 2.5× at $P_m = 15.1\%$ (Figure 5, bottom left, green line).
  • Test accuracy improves by up to 3.3 percentage points at $P_m = 26.4\%$ (from approximately 79.1% to roughly 82.4%).

Across all three architectures, test accuracy at the final training iteration (Figure 5, bottom right) shows that winning tickets maintain higher accuracy than the original network even after training accuracy reaches 100%, with the gap persisting to $P_m \approx 2\%$. The pattern is consistent: moderate pruning makes the network faster to train, more accurate, and better at generalizing.

Random reinitialization (Section 3, Figure 5, dashed lines). As with Lenet, randomly reinitialized versions of the same winning ticket architectures perform substantially worse:

• Reinitialized networks learn increasingly slower with continued pruning (dashed lines rise in the left panels of Figure 5), opposite to the winning tickets which learn increasingly faster (solid lines decrease).
• Test accuracy drops off more quickly for reinitialized networks than for winning tickets.
• However, unlike Lenet, Conv-2 and Conv-4 show an interesting exception: at moderate pruning levels, reinitialized networks maintain or even slightly improve test accuracy over the original (Figure 5, middle panels, dashed blue and orange lines initially flat or slightly rising). This indicates that at moderate sparsity, the structure of the winning ticket alone (without the specific initialization) can provide some benefit — consistent with Liu et al.'s (2019) observation that pruned architectures can be retrained from scratch. But this benefit disappears at higher sparsity, where only the original initialization maintains accuracy.

Dropout interaction (Section 3, Figure 6). When Conv-2, Conv-4, and Conv-6 are trained with dropout (rate 0.5) and then iteratively pruned:

• Dropout improves initial test accuracy: +2.1 percentage points for Conv-2, +3.0 for Conv-4, +2.4 for Conv-6 (Figure 6, right panel, dashed vs. solid lines at $P_m = 100\%$).
• Iterative pruning further improves accuracy on top of dropout: up to an additional +2.3 percentage points for Conv-2, +4.6 for Conv-4, and +4.7 for Conv-6.
• Learning becomes faster with iterative pruning as before, though less dramatically for Conv-2.
• The fact that winning tickets are still found under dropout regularization, and that the accuracy improvements from dropout and pruning are complementary (they stack), suggests these mechanisms operate through different channels — dropout does not subsume or replace the winning ticket phenomenon.

---

Deep Networks: VGG-19 and Resnet-18 on CIFAR10

Learning rate sensitivity (Section 4, Figures 7 and 8). The headline finding for both VGG-19 and Resnet-18 is that iterative pruning's ability to find winning tickets is critically dependent on the learning rate:

• At learning rate 0.1 (the standard rate used in the literature for both architectures), iterative pruning does not find winning tickets for either network. For VGG-19 (Figure 7, left panels, solid blue vs. dashed blue), the pruned networks with original initialization perform no better than randomly reinitialized networks — both degrade similarly as sparsity increases. For Resnet-18 (Figure 8, left panels), the same holds.

• At learning rate 0.01, the lottery ticket pattern reemerges. For VGG-19 (Figure 7, orange lines), winning tickets remain within 1 percentage point of the original accuracy when $P_m \geq 3.5\%$, whereas reinitialized networks lose accuracy rapidly. However, these winning tickets' accuracy falls short of the 0.1-rate unpruned network's accuracy. For Resnet-18 (Figure 8, orange lines), winning tickets reach up to 89.5% test accuracy when $41.7\% \geq P_m \geq 21.9\%$, close to but still below the 0.1-rate unpruned network's 90.5%.

• The crucial negative result at rate 0.1: the curves for original initialization and random reinitialization overlap almost completely for both architectures (Figures 7 and 8, blue and dashed blue lines). This means that at high learning rates, the initialization lottery is effectively not happening — the pruning procedure cannot distinguish between the original and random initializations because the aggressive optimization dynamics of SGD at rate 0.1 overwrite whatever signal the initialization provided. This is not a minor degradation; it is a qualitative absence of the lottery ticket phenomenon.

Learning rate warmup (Section 4, Figures 7 and 8, green lines). Introducing linear learning rate warmup (learning rate increases linearly from 0 to the target over $k$ iterations) bridges the gap:

• For VGG-19 (Figure 7, green lines): warmup with $k = 10{,}000$ iterations at learning rate 0.1 improves the unpruned network's test accuracy by about 1 percentage point (to roughly 91.5% at iteration 112K). More importantly, it restores the lottery ticket phenomenon: winning tickets exceed this improved accuracy when $P_m \geq 1.5\%$, and they consistently outperform randomly reinitialized versions (which follow the typical degradation pattern).

• For Resnet-18 (Figure 8, green lines): warmup with $k = 20{,}000$ at learning rate 0.03 enables winning tickets to match the 90.5% test accuracy of the 0.1-rate unpruned network (matching the gold standard that 0.01-rate alone could not reach) while still finding winning tickets when $P_m \geq 11.8\%$. The winning tickets at intermediate sparsity levels (e.g., $P_m = 27.1\%$) achieve accuracy comparable to or slightly exceeding the 0.1-rate unpruned network. The authors note they could not find hyperparameters enabling winning ticket discovery at the full 0.1 learning rate for Resnet-18 — 0.03 was the highest viable rate even with warmup.

• At lower budgets (earlier in training), the warmup-trained winning tickets initially appear to learn faster than the 0.1-rate unpruned network (Figures 7 and 8, left columns at 30K/10K iterations), but this advantage diminishes later in training as the staged learning rate drops take effect in the baseline.

Global vs. layer-wise pruning (Appendix I.1, Figures 38 and 39). For both VGG-19 and Resnet-18, global pruning (removing the lowest-magnitude weights across all convolutional layers collectively) finds smaller winning tickets than layer-wise pruning (removing a fixed percentage from each layer independently):

• For VGG-19 with warmup: global pruning finds winning tickets when $P_m \geq 1.5\%$, while layer-wise pruning only finds them when $P_m \geq 6.9\%$ (Figure 38, solid green vs. dashed green). This is a substantial difference — global pruning pushes the viable sparsity limit more than 4× further.
• For Resnet-18, the difference is less extreme but still present: global pruning supports somewhat smaller winning tickets than layer-wise pruning at the same accuracy level.

---

Ablation Studies and Robustness Checks

Iterative pruning strategy: resetting vs. continued training (Appendix B, Figures 9 and 10). Strategy 1 (resetting weights to original initialization after each pruning round) consistently outperforms Strategy 2 (continuing training from current weights without resetting). For Lenet (Figure 9), Strategy 1 maintains higher validation accuracy and faster early-stopping times to smaller network sizes. For Conv-2/4/6 (Figure 10), the same pattern holds across all three architectures. This ablation confirms that the resetting step — not just iterative refinement of the mask — is critical for finding the smallest, most effective winning tickets. Continued training (Strategy 2) is closer to standard pruning practice but produces masks that are less dependent on the original initialization and thus less informative about the lottery ticket hypothesis.

Pruning rate (Appendix G.4, Figure 29). For Lenet on MNIST, the iterative pruning rate (percentage of surviving weights removed per round) affects outcomes: lower rates (0.1 and 0.2, i.e., 10% and 20% per round) maintain higher validation accuracy and faster early-stopping to smaller network sizes compared to higher rates (0.4, 0.6, 0.8). The paper selects 0.2 (20% per round) as a compromise: it maintains much of the accuracy and speed of the 0.1 rate while requiring fewer rounds to reach a given sparsity. This suggests that pruning too aggressively in each round (removing too many weights at once) discards connections that would have proven useful if given another round of training to demonstrate their importance — an important detail for understanding why iterative pruning outperforms one-shot pruning.

Layer-wise pruning: convolutional vs. fully-connected layers (Appendix H.6, Figure 37). For Conv-2/4/6, pruning only convolutional layers leads to higher test accuracy and faster learning, while pruning only fully-connected layers degrades both. However, pruning convolutions alone has limited ability to reduce overall parameter count (convolutions constitute 1%, 11%, and 65% of parameters for Conv-2, Conv-4, and Conv-6 respectively). The optimal pruning rates differ per architecture — convolutional pruning rates of 10% (Conv-2/4) or 15% (Conv-6) per round are selected to balance the contributions of both layer types.

Pruning at iteration 0 (Appendix F.4, Figure 18). Pruning based on the initial weight magnitudes (before any training) and then training the resulting sparse network from scratch performs worse than random reinitialization of winning tickets. This demonstrates that the information about which weights are important cannot be extracted from initial magnitudes alone — the training process is necessary to identify which connections form the winning ticket. This is a critical negative result: it shows that winning tickets are not simply networks that happen to have large initial weights, and that the iterative training-pruning cycle genuinely discovers something that initial weight magnitudes do not reveal.

Random reinitialization vs. random sparsity (Appendix E, Figure 14). For Lenet on MNIST, randomly reinitialized winning tickets outperform random sparse subnetworks (same number of parameters but random connectivity). However, for all convolutional architectures, there is no significant difference between the two — the structure alone, even when randomly reinitialized, does not outperform a random sparse network of the same size. This implies that for convolutional networks, the mask $m$ discovered by pruning does not encode a useful inductive bias by itself — it only becomes beneficial when combined with the original initialization. For fully-connected networks on MNIST, the mask alone may provide some benefit (perhaps by pruning connections to uninformative input pixels, which are concentrated at the image edges in centered MNIST digits), but even there, the initialization provides substantial additional benefit.

Winning ticket initialization distributions (Appendix F.1–F.2, Figures 15 and 16). For Lenet trained with Adam (Figure 15), the distribution of winning ticket initializations becomes increasingly bimodal with pruning for the second hidden layer and output layer, with peaks on either side of zero. The peaks are asymmetric: the second hidden layer retains more positive initializations, the output layer more negative. The first hidden layer maintains its original distribution. For SGD-trained networks (Figure 16), the bimodality appears across all layers. In both cases, connections with higher-magnitude initializations are more likely to survive pruning — magnitude at the end of training correlates with magnitude at initialization, which is the mechanism by which magnitude-based pruning "selects" winning ticket connections. The asymmetry in the bimodal distributions suggests that the sign of the initialization may interact with the learning dynamics in a task-specific way.

Reinitializing from winning ticket distribution (Appendix F.3, Figure 17). When winning tickets are reinitialized by sampling from the empirical distribution of surviving initializations (preserving the bimodal shape, higher variance, and asymmetry) rather than from the original Gaussian, performance is only marginally better than random reinitialization from the Gaussian. The winning ticket with original initializations far outperforms both. This demonstrates that the exact values matter — the specific configuration of initial weights in the winning ticket cannot be reduced to a set of per-layer statistical properties.

Adding noise to winning tickets (Appendix F.7, Figure 25). Winning ticket performance degrades progressively as Gaussian noise is added to the initializations, but the tickets are surprisingly robust. Noise with standard deviation $0.5\sigma$ (where $\sigma$ is the standard deviation of the original Glorot initialization for that layer) barely changes accuracy or early-stopping time. As noise increases to $\sigma$, $2\sigma$, and $3\sigma$, accuracy decreases and learning slows, but even at $3\sigma$, the winning tickets outperform the randomly reinitialized networks. This suggests that winning ticket initializations occupy a region of weight space that is robust to perturbation but distinct from the distribution from which random initializations are drawn — a "basin" of effective initial configurations rather than a single point.

Network size scaling (Appendix G.6, Figure 31). When Lenet architectures with different initial sizes (from 25/9 to 900/300 hidden units, maintaining a 3:1 ratio) are pruned to the same absolute number of surviving weights, winning tickets from initially larger networks achieve higher test accuracy. The lines in Figure 31 (right) are approximately ordered from bottom to top by increasing initial network size. This means that the benefit of initial overparameterization persists even after pruning to identical parameter counts — larger initial networks contain better winning tickets, not just more of them.

Learning rate sweeps (Appendices G.2, H.2, I.4). For Lenet with Adam (Figure 26), the lottery ticket pattern (faster learning, maintained or improved accuracy with pruning) appears across all learning rates from $2.5 \times 10^{-5}$ to $3.2 \times 10^{-3}$, though the fastest rates produce the most pronounced benefits. For Conv-2/4/6 with Adam (Figure 32), the pattern holds across most learning rates, failing only at the very highest (e.g., $8 \times 10^{-4}$ for Conv-2), where early-stopping times decrease only briefly or not at all — presaging the learning rate sensitivity seen in deeper networks. For Resnet-18 and VGG-19 (Figures 42 and 43), rates above 0.01 consistently fail to produce winning tickets, confirming that the sensitivity is specifically a property of deeper architectures with SGD+momentum optimization.

Warmup iteration count (Appendix I.5, Figures 44 and 45). For VGG-19 at rate 0.1, accuracy of winning tickets improves rapidly as the warmup period $k$ increases from 0 to 5,000, with diminishing returns above $k = 5{,}000$. The paper selects $k = 10{,}000$ for VGG-19 and $k = 20{,}000$ for Resnet-18, both in the diminishing-returns regime. The precise value matters less than the qualitative necessity: some warmup is required, and more is generally better up to a point, but the effect saturates.

Dropout learning rates (Appendix H.5, Figure 36). Adding dropout requires re-tuning learning rates. For Conv-2, learning rate $3 \times 10^{-4}$ was selected because lower rates ($1 \times 10^{-4}$, $2 \times 10^{-4}$) caused some or all trials to fail to learn productively until several rounds of pruning had occurred. This is notable: networks that cannot learn at a given learning rate in early rounds become able to learn after pruning, suggesting pruning removes weights that were destabilizing optimization. For Conv-4 and Conv-6, learning rate $2 \times 10^{-4}$ provided the best combination of high initial accuracy and sustained lottery ticket benefits.

Weight movement during training (Appendix F.5, Figures 19 and 20). Weights that survive pruning (i.e., become part of the winning ticket) change by a larger magnitude between initialization and final training than weights that do not survive. They are also more likely to move away from zero (increase in magnitude). This contradicts the hypothesis that winning ticket weights are already near their optimal values at initialization — they are, if anything, the weights that undergo the most change during training. The authors interpret this as evidence that winning ticket initializations are "well placed in the optimization landscape for gradient descent to optimize productively," not that they are pre-trained or close to convergence.

Winning ticket connectivity (Appendix F.6, Figures 21–24). For Lenet trained with Adam, incoming connections to each unit are pruned approximately uniformly — each unit retains a number of incoming connections proportional to the overall pruning level. However, outgoing connections show a highly uneven distribution: some input units retain far more outgoing connections than others, consistent with the observation that MNIST digits are centered, making edge pixels less informative. Interestingly, Adam-trained winning tickets develop a much more uneven distribution of outgoing connectivity than SGD-trained ones, suggesting the optimizer influences which structural patterns emerge.

---

Critical Assessment

Does the lottery ticket hypothesis hold as stated? The experiments demonstrate that for all architectures tested (Lenet, Conv-2/4/6, VGG-19, Resnet-18), there exist sparse subnetworks that — when trained in isolation with their original initializations — reach test accuracy comparable to the original network in comparable or fewer training iterations. This is the core of the hypothesis, and the evidence is consistent across architectures, datasets, and optimizers (Adam, SGD, SGD+momentum). Specifically, winning tickets are found that meet all three criteria ($j' \leq j$, $a' \geq a$, $\|m\|_0 \ll |\theta|$) at sizes of 10–20% (or less) of the original network for every architecture studied.

However, there are important boundary conditions that the hypothesis statement does not capture but the experiments reveal:

1. The hypothesis does not hold at all learning rates. On VGG-19 and Resnet-18 at the standard learning rate of 0.1, the iterative pruning procedure fails to identify winning tickets — the lottery ticket phenomenon is absent (Figures 7 and 8, rate 0.1 panels). The hypothesis statement "a randomly-initialized, dense neural network contains a subnetwork..." is not technically falsified by this (the subnetwork might still exist even if the discovery procedure fails), but it raises the question of whether the hypothesis is a property of the network or a property of the network-plus-training-regime. The warmup experiments partially address this by showing that the phenomenon can be restored by modifying the optimization, but the limitation is genuine: winning tickets are not findable under all reasonable training conditions.

2. The definition of "commensurate accuracy" ($a' \geq a$) is met only down to a certain sparsity. For all architectures, there is a point beyond which winning tickets no longer match the original network's accuracy. For Lenet, this is $P_m \approx 3.6\%$; for VGG-19 with warmup, it is $P_m \approx 1.5\%$. The hypothesis does not claim there is no sparsity limit, but the existence of the limit means that the "winning ticket" is not arbitrarily sparse — there is a minimum parameter count below which even the best initialization cannot compensate.

3. The hypothesis is tested only on vision classification tasks with small datasets. The paper explicitly acknowledges this (Section 6): "We only consider vision-centric classification tasks on smaller datasets (MNIST, CIFAR10). We do not investigate larger datasets (namely Imagenet)." This is a significant scope limitation. Whether winning tickets exist in networks trained on ImageNet, in NLP tasks, in reinforcement learning, or in generative models is unknown from this paper.

Does initialization matter, or just structure? The random reinitialization experiments provide strong evidence that initialization matters — for every architecture, randomly reinitialized winning tickets perform substantially worse than those with original initializations (e.g., Figure 4a: 2.51× slower early-stopping at $P_m = 21.1\%$ for Lenet; Figure 5: dashed lines consistently above solid lines for early-stopping). The additional experiments in Appendix F (reinitializing from the empirical distribution does not help, adding noise progressively degrades performance) reinforce this conclusion.

However, the evidence is not completely one-sided. For Conv-2 and Conv-4 at moderate sparsity, randomly reinitialized networks do maintain or improve test accuracy (Figure 5, middle panels, dashed lines at high $P_m$). This is consistent with Liu et al. (2019) and suggests that at moderate sparsity, structure alone can suffice — the overparameterization of the original network is so extreme that even a random initialization of a moderately pruned architecture works. The initialization lottery becomes essential only when sparsity exceeds some threshold (roughly $P_m < 20\%$ for these architectures). The paper's claim that "initialization matters" is thus accurate but should be qualified: initialization is necessary for extreme sparsity, but optional for moderate sparsity where architecture alone provides enough inductive bias.

Are the winning tickets genuinely "found" by pruning, or created by it? This is a subtle philosophical question the paper acknowledges but cannot definitively answer. The iterative pruning procedure selects connections based on their trained magnitudes, but this selection is influenced by the random seed, the optimization trajectory, and the pruning schedule. Could a different mask, not discoverable by this procedure, also be a winning ticket? The paper has no way of testing this — it only studies the masks produced by magnitude-based pruning. The fact that one-shot pruning finds different (and worse) masks than iterative pruning suggests that the discovery procedure does not uniquely identify the "true" winning ticket.

Moreover, there is a potential circularity: magnitude-based pruning selects connections whose trained weights have high magnitude. If the correlation between trained magnitude and initialization quality is imperfect (which the paper's own data suggests — Figure 19 shows winning ticket weights change substantially during training), then magnitude-based pruning may miss genuine winning tickets and may include connections that are not part of any particularly good subnetwork. The paper's claim that "a standard pruning technique naturally uncovers such trainable subnetworks" is accurate but should be read as "this particular pruning technique finds some subnetworks that happen to be trainable" — not that it finds the optimal or only such subnetwork.

The generalization improvement claim is real but its source is ambiguous. Winning tickets consistently achieve higher test accuracy than the original network while matching training accuracy (which reaches ~100% in most cases — Appendix D, Figures 12 and 13). The paper attributes this to improved generalization, citing the Occam's Hill concept. However, an alternative explanation is that the original network underfits due to optimization difficulty (too many parameters relative to the gradient signal), and pruning removes weights that were impeding optimization, allowing the network to converge to a better solution. The paper's own data partially supports this: training accuracy does reach 100% for both winning tickets and reinitialized networks at most sparsity levels, but the test accuracy gap persists — which does point to generalization rather than optimization. But without a careful double-descent or bias-variance analysis, the precise mechanism by which winning tickets generalize better remains unclear.

Where are the negative results hidden? The paper is admirably transparent about negative results in several places, but some qualifications deserve emphasis:

• Resnet-18 at rate 0.1 never yields winning tickets even with warmup — the best warmup result uses rate 0.03 (Section 4). The paper presents this as a limitation, but it also means that for perhaps the most practically relevant architecture+hyperparameter combination, the lottery ticket phenomenon does not manifest.
• The "pruning at iteration 0" experiment is a decisive negative (Appendix F.4): taking initial magnitudes as the pruning criterion fails completely. This means winning ticket discovery genuinely requires training — you cannot shortcut it.
• The ReST-style revision model failure (if this paper had an analogous result) — actually, this paper does not have reinforcement learning experiments, but the conv-6 results show that at the highest pruning levels, even winning tickets degrade, and the paper does not hide this.
• One-shot pruning is strictly worse than iterative (Figure 4). The paper does not oversell one-shot pruning as a practical alternative; it uses the comparison to motivate why iterative pruning is necessary for finding the smallest tickets.

What experiments would strengthen the paper? Several gaps are evident:

1. No ImageNet experiments. The computational expense is the stated reason, but without larger-scale experiments, it's unknown whether winning tickets exist at scale. Subsequent work (Frankle et al., 2020, "Linear Mode Connectivity and the Lottery Ticket Hypothesis") partially addressed this, but within this paper it remains a limitation.
2. No test of whether winning tickets transfer across tasks. If a winning ticket discovered for CIFAR10 could be trained on a different dataset (e.g., CIFAR100, or even MNIST with appropriate input layer adaptation), that would strongly support the claim that winning tickets have a "generic" property making them trainable, rather than being tuned to the specific task. The paper does not attempt this.
3. No comparison of winning tickets found by different pruning criteria. Magnitude pruning is only one of many pruning heuristics (gradient-based, second-order, activation-based, etc.). If winning tickets are a real phenomenon, they should be discoverable (perhaps at different sparsity levels) by other pruning methods — testing this would distinguish between "magnitude pruning happens to find trainable subnetworks" and "trainable subnetworks exist and can be found by various methods."
4. No measurement of the overlap between winning tickets found from different random seeds. Do five different trials of the lottery ticket experiment on the same architecture produce similar masks? Figure 21–24 shows connectivity distributions that are consistent across trials, but the paper does not quantify mask overlap (e.g., Intersection over Union). High overlap would suggest a specific winning ticket is "the" right answer; low overlap would suggest many different winning tickets exist.
5. The test set for CIFAR10 is 10,000 examples, the validation set is 5,000, and the test accuracy differences for convolutional architectures are on the order of 3-4 percentage points (e.g., Conv-2: 67.6% → 71.0%). The paper reports only min/max error bars across five trials, not confidence intervals or standard deviations. For differences of ~3.4 percentage points on a 10,000-example test set with five trials, statistical significance is not formally established — the claim that winning tickets "reach higher test accuracy" would be strengthened by proper statistical testing.

6. Limitations and Trade-offs

The Discovery Cost Is So High It Precludes Practical Deployment

The assumption or constraint. The iterative pruning procedure used to find winning tickets requires training the network 15 or more times consecutively — once per pruning round — and repeating this for each independent trial (five trials for most experiments). The paper is transparent about this cost, stating directly:

"iterative pruning is computationally intensive, requiring training a network 15 or more times consecutively for multiple trials. In future work, we intend to explore more efficient methods for finding winning tickets that will make it possible to study the lottery ticket hypothesis in more resource-intensive settings." (Section 6)

The one-shot pruning alternative (Section 2, Figure 4c) reduces this to a single training run but produces larger, less effective winning tickets — the smallest viable one-shot ticket for Lenet ($P_m \approx 5\%$) is substantially larger than the smallest iterative ticket ($P_m \approx 3.6\%$), and at comparable sparsity, iterative tickets learn faster and achieve higher accuracy.

The consequence. The lottery ticket hypothesis, as demonstrated, is a scientific discovery about what exists — not a practical recipe for training networks. A practitioner who reads the headline result ("winning tickets are 10–20% of the original size and train faster") might reasonably hope to deploy this to reduce training costs. But the discovery procedure costs more than training the original dense network — to find a winning ticket at $P_m = 3.6\%$ for Lenet requires approximately 15 rounds of training at 50,000 iterations each (750,000 total training iterations across rounds), compared to 50,000 for the original network. This is a ~15× increase in total computation, not a savings. The efficiency gains from the winning ticket (38% faster early-stopping, 0.3 percentage points higher accuracy) apply only after the ticket has been found — they do not amortize the discovery cost, and the paper does not attempt to calculate a break-even point where the combined discovery + training cost becomes favorable compared to simply training the dense network.

For deeper networks, the situation is worse. Resnet-18 requires 30,000 iterations per round, and the paper explores warmup, multiple learning rates, and pruning schedules — the total computational burden to find a single winning ticket configuration for Resnet-18 is enormous relative to simply training the network once at rate 0.1.

What evidence exists in the paper. The iterative pruning procedure is described in Section 1 and Appendix B. The number of rounds is implicitly defined by $P_{\text{final}} = P_{\text{initial}} \times (1 - s)^n$, where $s = 0.2$ per round. To reach $P_m = 3.6\%$ from 100% requires roughly $\log(0.036) / \log(0.8) \approx 15$ rounds. Each round trains for the full iteration budget (50,000 for Lenet, 20,000–30,000 for Conv-2/4/6). The one-shot comparison (Figure 4c) quantifies what is lost by reducing to a single round: one-shot tickets have higher early-stopping iteration and lower test accuracy at the same sparsity, and they cannot be pruned as far before failing. The paper does not report total FLOPs or GPU-hours for any experiment, nor does it attempt to compare the total cost of winning ticket discovery against alternative ways of achieving the same accuracy (e.g., training a wider dense network, or training longer with better hyperparameters).

Mitigation status. The paper does not attempt to mitigate this cost — it explicitly defers efficient discovery to future work (Section 6). The one-shot pruning variant is explored as a partial mitigation but is acknowledged as insufficient for finding the smallest tickets. The authors note that "we intend to explore more efficient methods for finding winning tickets," including structured pruning and non-magnitude pruning methods, but none are developed or tested here. The limitation is inherent to the paper's framing as a discovery paper rather than a methods paper, but it fundamentally constrains the practical relevance of the findings.

---

The Lottery Ticket Phenomenon Is Not Robust to Standard High Learning Rates on Deep Networks

The assumption or constraint. On VGG-19 and Resnet-18 — the architectures most representative of practical deep learning — iterative pruning fails to find winning tickets at the standard learning rate of 0.1 (used in the literature for both architectures). The paper explicitly reports this. For VGG-19 at rate 0.1 (Figure 7, left panels, blue vs. dashed blue):

"At the higher learning rate, iterative pruning does not find winning tickets, and performance is no better than when the pruned networks are randomly reinitialized."

For Resnet-18 at rate 0.1 (Figure 8, left panels), the same holds: the curves for original initialization and random reinitialization overlap almost completely. Even with warmup, the authors "could not find hyperparameters for which we could identify winning tickets at the original learning rate, 0.1" for Resnet-18 — the highest viable rate was 0.03 (Figure 8, green line).

The lower learning rate (0.01) restores the lottery ticket pattern, but at a cost: the winning tickets at rate 0.01 achieve lower absolute accuracy than the unpruned network trained at rate 0.1. For Resnet-18, the best rate-0.01 winning tickets reach 89.5% test accuracy, compared to 90.5% for the rate-0.1 unpruned network. Warmup partially bridges this gap — Resnet-18 at rate 0.03 with warmup achieves 90.5% — but only by moving to a higher (but still sub-0.1) learning rate.

The consequence. The lottery ticket hypothesis is not a property of neural networks in isolation — it is a property of neural networks paired with sufficiently gentle optimization dynamics. At the aggressive learning rates that produce the best results on these architectures, the iterative pruning procedure cannot distinguish between original and random initializations. This means that for a practitioner using standard hyperparameters (rate 0.1 with momentum), winning tickets are effectively invisible — the procedure that discovers them fails, and even if a winning ticket objectively exists within the network, there is no known way to extract it without sacrificing final accuracy or switching to a suboptimal learning rate.

This is a more severe constraint than the discovery cost alone. Even if a practitioner were willing to pay the 15× computational premium to find a winning ticket, they might not be able to do so without changing their training setup in ways that reduce the final accuracy of the unpruned baseline. The warmup approach offers a partial resolution, but (a) it requires hyperparameter tuning (the warmup duration $k$ must be selected; Figures 44 and 45 show accuracy depends on $k$), and (b) for Resnet-18, it still falls short of the standard 0.1 rate. The phenomenon thus appears to be contingent on optimization details that are not well understood and that the paper does not fully characterize.

What evidence exists in the paper. The core evidence is in Figures 7 and 8 (Section 4), where the rate-0.1 panels show near-perfect overlap between winning ticket and random reinitialization curves, while the rate-0.01 and warmup panels show clear separation. Appendix I.4 (Figures 42 and 43) sweeps additional learning rates for VGG-19 and Resnet-18, confirming that rates above 0.01 consistently fail. The warmup experiments (Figures 44 and 45) show that $k$ must be chosen appropriately — accuracy improves rapidly with $k$ up to ~5,000 iterations and then saturates, but the optimal $k$ differs between architectures (10,000 for VGG-19, 20,000 for Resnet-18). The paper does not provide a theoretical explanation for why learning rate sensitivity occurs, only the empirical observation and the warmup workaround.

Mitigation status. Warmup is the paper's primary mitigation, and it is partially effective — it enables winning ticket discovery at higher learning rates than would otherwise be possible. However, it does not fully resolve the limitation: Resnet-18 at its standard rate of 0.1 remains inaccessible even with warmup. The paper identifies this as an open problem, stating in Section 6: "In future work, we plan to explore why warmup is necessary and whether other improvements to our scheme for identifying winning tickets could obviate the need for these hyperparameter modifications." No mechanistic explanation for the sensitivity is offered, and no alternative mitigation (beyond warmup and learning rate reduction) is tested.

---

The Paper Studies Only Small Vision Benchmarks, Leaving Generality Unestablished

The assumption or constraint. All experiments are confined to MNIST (grayscale 28×28 digit classification) and CIFAR10 (color 32×32 object classification). The paper explicitly acknowledges this scope limitation:

"We only consider vision-centric classification tasks on smaller datasets (MNIST, CIFAR10). We do not investigate larger datasets (namely Imagenet (Russakovsky et al., 2015)): iterative pruning is computationally intensive, requiring training a network 15 or more times consecutively for multiple trials." (Section 6)

The architectures studied span a meaningful range (fully-connected, shallow convolutional, deep convolutional with batchnorm/residuals), but they are all vision classifiers trained on datasets with 10 output classes and 50,000–60,000 training examples. The paper does not experiment with NLP tasks, speech recognition, reinforcement learning, generative modeling, or any non-classification objective. It does not study transfer learning, multi-task learning, or any setting where the winning ticket for one task might be evaluated on another.

The consequence. It is unknown whether winning tickets exist in (a) larger-scale vision tasks where the training dynamics and overparameterization ratios are different, (b) non-vision domains with different data modalities and architectural conventions, or (c) tasks where "accuracy" is not the sole metric of interest (e.g., calibration, robustness, fairness). ImageNet-scale experiments are a particularly significant gap: ImageNet has 1.2 million training images across 1,000 classes, and the overparameterization ratio (parameters per training example) is very different from CIFAR10. If winning tickets fail to exist at ImageNet scale — perhaps because the optimization landscape is different, or because the "winning ticket" sparsity level would be impractically close to 100% — then the phenomenon would be a curiosity of small-scale training rather than a general principle.

The architectural diversity within the paper is meaningful but incomplete. The paper studies VGG-style and ResNet-style convolutional networks, but does not study transformers, recurrent networks, graph neural networks, or any architecture without a strong spatial locality prior. The lottery ticket hypothesis as stated applies to "dense, randomly-initialized, feed-forward networks" — but whether the convolutional inductive bias interacts with or partially explains the existence of winning tickets is untested.

What evidence exists in the paper. The paper studies six architectures across two datasets, spanning fully-connected (Lenet), shallow-to-moderate convolutional (Conv-2/4/6), and deep convolutional with modern techniques (VGG-19, Resnet-18). This is broader than many contemporaneous pruning papers, but it does not constitute a systematic sweep across task modalities, data scales, or architecture families. The paper reports no experiments outside of supervised image classification. The computational expense of iterative pruning on ImageNet is cited as the barrier (Section 6), but no abbreviated or approximate experiment (e.g., pruning on a subset of ImageNet, or using a smaller ImageNet-scale model) is attempted.

Mitigation status. The paper acknowledges this as a limitation and states intent to explore larger datasets in future work. It does not attempt to extrapolate from the small-scale results to predict what would happen at larger scale, nor does it discuss whether any theoretical considerations would suggest the phenomenon should or should not generalize. The limitation is essentially unmitigated — the reader is left to decide whether to believe the lottery ticket hypothesis generalizes based on the consistency of results across the studied architectures and datasets, but that consistency is within a narrow domain.

---

The Winning Ticket Definition Permits Degraded Performance Below a Sparsity Threshold, and That Threshold Varies Wildly Across Architectures

The assumption or constraint. The lottery ticket hypothesis requires that a winning ticket satisfy $a' \geq a$ (accuracy at least as good as the original) and $j' \leq j$ (training at most as long as the original). But for every architecture studied, there exists a sparsity level below which these conditions no longer hold. This failure point varies dramatically:

• For Lenet on MNIST: winning tickets fail below $P_m \approx 3.6\%$ (Figure 4a, blue line drops below original accuracy).
• For VGG-19 with warmup: winning tickets viable down to $P_m \approx 1.5\%$ (Figure 7, green line).
• For Conv-2: viable down to $P_m \approx 2\%$ (Figure 5, blue line).
• For Resnet-18 with warmup: viable down to $P_m \approx 11.8\%$ (Figure 8, green line).

These thresholds are not predicted by any model — they are simply the empirical points at which the curves cross the original accuracy line. The paper offers no explanation for why Resnet-18's winning tickets are viable only to ~12% of original weights while VGG-19's are viable to ~1.5%, despite both being deep convolutional networks trained on CIFAR10. The variation spans nearly an order of magnitude (1.5% vs. 11.8%).

The consequence. The lottery ticket hypothesis states that a winning ticket exists with $\|m\|_0 \ll |\theta|$, but it does not specify how much smaller, and the empirical answer varies substantially. A practitioner who reads the headline result ("winning tickets are less than 10–20% of the original size") and expects this to hold for their own architecture may be disappointed if their network sits at the unfavorable end of the spectrum. The paper's claim of "10–20%" is a rough summary that happens to exclude Resnet-18's threshold (~12%, barely within the range if rounding generously) and overstates the compression achievable on some architectures.

More fundamentally, the existence of a sharp failure threshold suggests that winning tickets are not arbitrarily small — there is a minimum parameter count below which the network cannot learn the task to the original accuracy, regardless of initialization quality. This minimum likely depends on the intrinsic difficulty of the task, the architecture's capacity, and the interaction between the two. The paper does not investigate what determines this threshold, leaving open the question of whether the "winning ticket" concept is fundamental (a specific subnetwork exists) or asymptotic (you can push sparsity until you hit the task's intrinsic capacity limit, and the initialization lottery just determines where within that regime you can go).

What evidence exists in the paper. Every figure tracking accuracy vs. $P_m$ shows accuracy eventually dropping below the original level. For Lenet: Figure 4a, blue line falls below original accuracy at $P_m < 3.6\%$. For Conv-2/4/6: Figure 5, solid lines fall below original test accuracy at $P_m < 2\%$ for all three architectures (bottom right panel). For VGG-19: Figure 7, green line ($P_m \geq 1.5\%$ viable). For Resnet-18: Figure 8, green line ($P_m \geq 11.8\%$ viable). The variation in failure thresholds across architectures is visible by comparing these figures but is never tabulated or discussed as a finding. The paper does not analyze what architectural properties (depth, width, residual connections, parameter counts, convolution-to-FC ratio) correlate with the minimum viable winning ticket size.

Mitigation status. The paper does not frame the sparsity threshold as a limitation — it presents the fact that winning tickets exist down to 10–20% of original size as a positive result, and the eventual accuracy decline as expected behavior (the Occam's Hill, where too much pruning hurts). However, the variation in thresholds and the lack of any predictive framework for estimating them constitutes an unaddressed gap. The paper does not discuss what "winning ticket at $P_m = 0.1\%$" experiments would look like or why they are absent. The implicit assumption is that the reader will accept that some sparsity limit exists and focus on the regime where winning tickets are found, but the threshold's architecture-dependence is a practical limitation for anyone hoping to apply these ideas to a new architecture.

---

Structure vs. Initialization: The Contribution of Architecture Alone Is Not Fully Resolved

The assumption or constraint. The paper's central claim is that initialization is essential to winning ticket success, demonstrated by the random reinitialization experiments. However, the evidence for this claim is not uniform across architectures and sparsity levels, and the paper does not fully reconcile its findings with Liu et al. (2019), who showed that randomly reinitialized pruned architectures can match original accuracy at moderate sparsity.

Specifically:

• For Conv-2 and Conv-4 at moderate sparsity, randomly reinitialized winning tickets maintain or slightly improve test accuracy over the original network (Figure 5, middle panels, dashed blue and orange lines at high $P_m$). The paper acknowledges this: "test accuracy at early-stopping time initially remains steady and even improves for Conv-2 and Conv-4, indicating that — at moderate levels of pruning — the structure of the winning tickets alone may lead to better accuracy."
• The comparison of randomly reinitialized winning tickets vs. random sparse subnetworks (Appendix E, Figure 14) shows that for all convolutional architectures, there is no significant difference between the two — the mask $m$ provides no structural advantage over a random mask when both are randomly initialized. Only for the fully-connected Lenet does the mask alone outperform random sparsity.

These observations complicate the narrative. The paper's core message is "initialization, not just structure, matters." But the data shows that (a) structure does matter at moderate sparsity for convolutional networks (it alone can match or exceed original accuracy), and (b) structure does not matter for convolutional networks when compared against random sparsity (a random mask does just as well as the pruned mask when both are randomly initialized). These two facts are in tension: if a randomly initialized pruned mask matches the original accuracy, but a randomly initialized random mask does not, then structure matters. If a randomly initialized random mask also matches the original accuracy (as Figure 14 suggests for convolutional networks), then neither structure nor initialization matters at moderate sparsity — you can just train a random sparse network of that size.

The consequence. The paper's claim that "initialization is crucial" is supported for extreme sparsity (where winning tickets outperform randomly reinitialized versions by large margins) but is not supported universally. At moderate sparsity on convolutional architectures, a practitioner could achieve similar results by randomly pruning a network and training from scratch — the winning ticket procedure provides no benefit in that regime. The regime where initialization matters is specifically the high-sparsity regime, which varies by architecture. This is a more nuanced and less general claim than the paper's headline suggests. The lottery ticket hypothesis as formally stated ($a' \geq a$, $j' \leq j$) is satisfied in both regimes, but the mechanism (initialization vs. architecture) changes depending on sparsity.

Furthermore, the finding that a randomly initialized random mask performs as well as a randomly initialized pruned mask for convolutional networks (Figure 14) raises the question of whether the pruning procedure is actually finding a special mask at all for these architectures, or whether any sufficiently sparse network trained from scratch would work. If the latter, then the lottery ticket hypothesis for convolutional networks reduces to the observation that sparse networks can be trained — not that a specific fortuitously-initialized subnetwork exists within the dense initialization.

What evidence exists in the paper. The key evidence is split across multiple figures. Figure 5 (dashed lines at high $P_m$) shows reinitialized Conv-2/4 outperforming the original accuracy. Appendix E, Figure 14 shows random reinitialization and random sparsity overlapping for all convolutional architectures. The paper addresses the Liu et al. conflict explicitly in Section 5 (Discussion): "Liu et al. (2019) find that pruned networks are indeed trainable when randomly reinitialized, seemingly contradicting conventional wisdom... Our experiments in Figure 7 confirm these findings at this level of sparsity (below which Liu et al. do not present data). However, after further pruning, initialization matters." This is a fair characterization but acknowledges that the paper's contribution is specifically about the high-sparsity regime where Liu et al. did not test.

Mitigation status. The paper partially addresses this by acknowledging the regime where structure alone works (Sections 3 and 5), but it does not systematically characterize the transition point between the "structure suffices" regime and the "initialization is essential" regime. There is no experiment that measures, for each architecture, the sparsity at which random reinitialization performance diverges from winning ticket performance. The paper also does not explain why structure alone helps for Lenet (random reinitialization > random sparsity) but not for convolutional networks (random reinitialization ≈ random sparsity) — the hypothesis that "only certain parts of the MNIST images contain useful information for classification, meaning connections in some parts of the network will be more valuable than others" (Appendix E) is plausible but untested. The boundary between the two regimes remains qualitative and post-hoc.

---

Sparse Pruning Produces Architectures Incompatible with Modern Hardware

The assumption or constraint. The paper uses unstructured magnitude-based pruning: each weight is pruned independently based on its individual magnitude, producing a sparse pattern where the surviving connections are scattered irregularly across the weight matrices. The resulting winning tickets are sparse in the sense of having fewer non-zero entries, but they do not have any regular structure (e.g., entire filters, channels, or blocks removed) that would make them efficient on contemporary hardware. The paper acknowledges this explicitly:

"Sparse pruning is our only method for finding winning tickets. Although we reduce parameter-counts, the resulting architectures are not optimized for modern libraries or hardware. In future work, we intend to study other pruning methods from the extensive contemporary literature, such as structured pruning (which would produce networks optimized for contemporary hardware) and non-magnitude pruning methods (which could produce smaller winning tickets or find them earlier)." (Section 6)

The consequence. Even if a practitioner were willing to pay the discovery cost and navigate the learning rate sensitivity, the resulting winning ticket would not actually run faster on GPUs or TPUs. Modern deep learning hardware and libraries (cuBLAS, cuDNN, Tensor Cores) are optimized for dense matrix multiplications on regular tensor shapes. Unstructured sparsity — where a matrix has, say, 80% of its entries zeroed out in an irregular pattern — does not translate to proportional speedup because the hardware cannot efficiently skip the zero entries without specialized sparse computation support (which, at the time of this paper and largely still today, provides limited practical acceleration compared to dense operations). Achieving real inference or training speedup requires structured sparsity — removing entire filters, channels, or blocks so that the remaining computation is still dense on smaller tensors.

This means the winning tickets demonstrated in the paper provide parameter-count reduction without corresponding computational speedup. A winning ticket at $P_m = 3.6\%$ uses 96.4% fewer parameters than the original network but would not run 27.8× faster (the reciprocal of 0.036) on standard hardware. The practical benefit of the lottery ticket hypothesis — the motivating question "why do we not train this smaller architecture instead in the interest of making training more efficient?" — is thus not realized by the unstructured winning tickets demonstrated here. The "efficiency" is in parameter count, not in wall-clock time or energy consumption.

This limitation also interacts with the discovery cost issue. If finding a winning ticket costs 15× the training time and the resulting ticket provides no wall-clock speedup at inference or for future training runs, then the entire exercise provides scientific insight but no practical compute savings whatsoever — the cost of discovery is pure overhead with no downstream efficiency dividend.

What evidence exists in the paper. The paper uses flat, per-weight magnitude pruning throughout — each weight is an independent pruning decision. Figure 21 (incoming connectivity per unit) and Figure 23 (outgoing connectivity per unit) show that surviving connections are distributed across units, not concentrated into dense sub-blocks. The paper does not measure inference speed, training speed, or GPU utilization for any winning ticket. It does not compare structured pruning (e.g., filter-level pruning as in Li et al., 2016) against unstructured pruning for winning ticket discovery. The limitation is acknowledged in Section 6 but not measured or quantified.

Mitigation status. The paper identifies structured pruning as future work but does not test it. There is no guarantee that structured pruning methods would also find winning tickets — it is possible that the "right" subnetwork for a given task is inherently unstructured, and forcing structural regularity would either increase the minimum viable size or make the winning ticket undiscoverable. The paper does not discuss this tension. The mitigation is purely aspirational: "in future work, we intend to study other pruning methods."

7. Implications and Future Directions

How This Work Changes the Landscape

The lottery ticket hypothesis reshapes how we think about neural network training at a fundamental level. Before this paper, the dominant view of pruning was retrospective: train a large network to learn a function, then compress away the redundant parts. The network starts as a blank slate and acquires structure through training. The lottery ticket hypothesis inverts this: the winning subnetwork was already present at initialization, and training serves merely to discover it. This is not an incremental refinement of pruning methodology — it is a reframing of what training itself accomplishes.

The concrete shift is from "training builds useful structure" to "training selects among pre-existing structures." The evidence for this inversion is the resetting step (Section 1, Step 4): retraining from the original initial weights produces a better, faster-learning network than retraining from trained weights or random weights on the same architecture. If training were constructing competence from nothing, resetting to original values would destroy it. The fact that resetting improves performance means the competence was latent in the initial configuration — training revealed which connections to keep but did not need to set their values to any particular trained state.

This reframing reconciles several previously contradictory observations:

The pruned-architecture trainability debate. Liu et al. (2019) showed that pruned architectures randomly reinitialized can match the original network at moderate sparsity, seemingly contradicting the conventional wisdom (from Li et al., 2016; Han et al., 2015) that pruned architectures are hard to train from scratch. The lottery ticket hypothesis resolves this by showing that both claims are true in different regimes. At moderate sparsity ($P_m \gtrsim 20\%$ for convolutional networks), the architecture alone encodes sufficient inductive bias — you can reinitialize randomly and still train successfully, exactly as Liu et al. found. At extreme sparsity ($P_m \lesssim 5\%$ for Lenet, $P_m \lesssim 2\%$ for Conv-2/4/6, $P_m \lesssim 1.5\%$ for VGG-19), the specific initialization becomes essential — random reinitialization fails dramatically (Figures 5 and 7). Liu et al. tested only the moderate regime; Han et al. and Li et al. observed the extreme regime. The controversy dissolves once sparsity level is accounted for.

The overparameterization puzzle. Why are larger networks easier to train? Prior theoretical work (Du et al., 2019) framed this in terms of optimization landscape smoothing — overparameterization makes the loss surface more benign. The lottery ticket hypothesis offers a complementary, mechanistic explanation: larger networks contain a combinatorially larger number of possible subnetworks, and SGD implicitly searches this space. The network size scaling experiment (Appendix G.6, Figure 31) directly supports this: when networks of different initial sizes are pruned to the same absolute parameter count, the ones derived from larger initial networks achieve higher accuracy. The benefit of overparameterization persists after pruning to identical sizes, meaning the larger initial network contained a qualitatively better subnetwork — not just smoother optimization dynamics during its own training.

The compression-generalization connection. Zhou et al. (2018) and Arora et al. (2018) proved tighter generalization bounds for networks that can be compressed further. The lottery ticket hypothesis gives this a mechanistic interpretation: compression (pruning) reveals a subnetwork that was already present at initialization, and that subnetwork's generalization properties were inherent to it from the start. The relationship is not causal (compression doesn't cause better generalization); rather, both compression tolerance and generalization quality are consequences of the same underlying structure — the fortuitously-initialized winning ticket. The Occam's Hill pattern (accuracy peaks at moderate sparsity, then declines — Figure 4a middle panel, Figure 5 bottom right) reflects the transition from a regime where pruning removes overfitting subnetworks (improving generalization) to a regime where it removes essential capacity (harming it).

This reframing makes certain research directions more attractive:

• Initialization scheme design becomes central. If winning tickets are latent in the initial weights, then better initialization schemes could increase the probability that a random initialization contains a high-quality winning ticket, or could make winning tickets discoverable at larger sparsity levels. The paper's finding that Glorot initialization supports winning tickets across architectures, while the specific initial values (not just their distribution) matter (Appendix F.3, Figure 17), suggests there is substantial room for targeted initialization research.

• Early winning ticket identification becomes the key practical goal. The discovery cost (15+ training cycles) is the paper's most severe practical limitation. If winning tickets could be identified early in training — or even at initialization — the computational premium of the lottery ticket approach would vanish, and training could be accelerated by training only the winning ticket from the start. The paper's finding that weights that will survive pruning diverge from other weights in their movement patterns during training (Appendix F.5, Figure 19) offers hope that such early identification is possible.

• Understanding optimization dynamics through the winning ticket lens. The learning rate sensitivity on deep networks (Section 4) implies that the interaction between optimization aggressiveness and subnetwork selection is non-trivial. If SGD with high learning rates fails to preserve the initialization signal that identifies winning tickets, then the choice of optimizer directly determines whether the lottery ticket phenomenon manifests. This opens a line of inquiry into why certain optimizers and learning rates allow subnetwork selection while others do not.

And makes other directions less attractive:

• Pruning as mere compression becomes less interesting as a scientific question. If the value of pruning is primarily in revealing pre-existing structure, then research focused solely on achieving higher compression ratios for inference — without reference to what the compression reveals about the network — addresses an engineering problem rather than a scientific one. The paper reframes pruning from a deployment tool to a diagnostic tool for studying network structure.

• Architecture search as pure structure optimization is challenged. The lottery ticket hypothesis implies that a network's trainability depends on the conjunction of architecture and initialization. Architecture search methods that optimize only connectivity patterns (e.g., evolutionary approaches that mutate masks) may miss the critical role of initialization, finding architectures that are structurally sensible but lack the fortuitous initial weight configurations that make them trainable. The paper's finding that randomly reinitialized winning tickets underperform original ones (Figures 4a, 5) directly demonstrates that architecture is necessary but not sufficient.

Follow-Up Research This Work Enables

Do winning tickets transfer across tasks, datasets, or architectures? The paper demonstrates that winning tickets exist for specific (architecture, dataset) pairs, but it never tests whether a winning ticket discovered for one task is also a winning ticket for another. A transfer experiment would determine whether winning tickets encode a generic property (e.g., a connectivity pattern that makes gradient descent effective regardless of the data distribution) or a task-specific one (e.g., an inductive bias tuned to the particular dataset). The strongest version of this experiment: identify a winning ticket for CIFAR10 on Resnet-18, then train that same mask (with its original initializations) on CIFAR100, STL-10, or a downsampled ImageNet. If the winning ticket transfers (trains faster and more accurately than a random sparse network of the same size on the new task), it suggests winning tickets capture something fundamental about optimization. If it does not transfer, winning tickets are task-specific structures, and the lottery ticket hypothesis must be conditioned on the training data as well as the architecture.

What is the overlap between winning tickets discovered from different random seeds? The paper runs five trials of each lottery ticket experiment and reports average performance, but it never analyzes whether the five resulting masks are similar or different. Computing the Intersection-over-Union (IoU) between pairs of winning ticket masks at the same sparsity level would answer a fundamental question: is there a single (or small set of) "canonical" winning ticket(s) that SGD reliably discovers, or are there many distinct winning tickets that all work? If overlap is high (IoU > 0.8), the winning ticket is a specific structure that is robust to random seed — suggesting it might correspond to a meaningful architectural motif. If overlap is low (IoU < 0.3), there are many functionally equivalent but structurally distinct winning tickets, and the lottery ticket hypothesis is about the existence of some well-initialized subnetwork, not about a specific one. The per-unit connectivity distributions in Appendix F.6 (Figures 21–24) hint at consistent structural patterns (input layer outgoing connectivity is bimodal, with edge pixels retaining fewer connections), but these are aggregate statistics; a direct intersection measurement would resolve whether individual connections are preserved across seeds.

Can winning tickets be identified early in training, or even at initialization, without full training? The computational cost of iterative pruning (15+ training cycles) is the paper's central practical limitation. The discovery cost would be substantially reduced if winning tickets could be identified early in the first training run. The paper provides suggestive evidence that this might be possible: Appendix F.5 (Figures 19 and 20) shows that weights that will eventually survive pruning can be distinguished from those that will not by their movement patterns during training — winning ticket weights change more in magnitude between initialization and convergence, not less. A concrete follow-up: monitor the gradient or weight-change statistics during the first $k$ training iterations (for small $k$, e.g., 500, 1,000, 5,000), and use these as a pruning criterion instead of final trained magnitudes. Compare the resulting winning tickets (size, accuracy, learning speed) against those found by full iterative pruning. If early-training statistics are sufficiently predictive, winning ticket discovery could be integrated into a single training run — prune progressively based on early movement patterns, never restarting from scratch.

Why does learning rate warmup restore winning ticket discovery on deep networks? The paper's most puzzling finding is that on VGG-19 and Resnet-18, iterative pruning fails to find winning tickets at the standard learning rate (0.1) but succeeds at lower rates (0.01) or with warmup (Section 4, Figures 7 and 8). The mechanism is not explained. A targeted follow-up would instrument the training dynamics at different learning rates to measure what differs: track the correlation between initial weight magnitude and final trained magnitude as a function of learning rate; measure the gradient norm and weight update magnitude in the first few hundred iterations with and without warmup; test whether the failure at high learning rates is due to large initial weight updates that "scramble" the initialization signal before it can be captured by magnitude-based pruning. A clean negative result would be: if high learning rate + cold start causes weights to move so far in the first few hundred iterations that their initial values become decorrelated from their eventual trained magnitudes, then magnitude pruning at the end of training cannot select for initial values — it's selecting for something else entirely. Warmup would fix this by keeping early weight updates small, preserving the initialization-to-trained-magnitude correlation. Testing this hypothesis directly would explain when the lottery ticket phenomenon should be expected to manifest and when it should not.

Do structured pruning methods also find winning tickets? The paper uses only unstructured magnitude pruning, producing winning tickets with irregular sparsity patterns that provide no hardware efficiency benefit (Section 6). The natural follow-up is to test whether structured pruning — removing entire filters, channels, or blocks based on aggregate statistics (e.g., filter L1 norm) — can also discover winning tickets. The specific experiment: apply structured pruning (e.g., filter-level pruning as in Li et al., 2016) iteratively with resetting, identically to the unstructured procedure, and measure whether the resulting structured winning ticket trains faster or more accurately than a randomly reinitialized structured-pruned network. If structured winning tickets exist, the lottery ticket hypothesis extends to practically useful sparse architectures. If they do not — if the fortuitous initialization requires fine-grained, per-weight selection — then the lottery ticket phenomenon may be fundamentally tied to unstructured sparsity, limiting its practical impact on hardware-efficient training. The paper's finding that pruning convolutions alone improves accuracy while pruning FC layers degrades it (Appendix H.6, Figure 37) hints that some structural granularities may be more compatible with winning ticket discovery than others.

Does the lottery ticket hypothesis hold at scale (ImageNet, transformers, large language models)? The paper's self-acknowledged most significant gap is the absence of experiments beyond small vision benchmarks. The computational expense of iterative pruning on ImageNet-scale models and datasets is severe, but a minimal viability experiment is feasible: take a standard ResNet-50 trained on ImageNet, perform one-shot pruning (not iterative, to keep cost manageable) at several sparsity levels with resetting, and test whether the resulting tickets train from scratch to match the original accuracy. Even a one-shot experiment at ImageNet scale would provide a crucial data point on whether the phenomenon persists when the overparameterization ratio (parameters per training example) is very different — ImageNet has ~1,000× more training examples than CIFAR10 and typical architectures have correspondingly more parameters, potentially changing the relationship between sparsity and trainability. More ambitiously, testing whether winning tickets exist in transformer architectures for NLP tasks (e.g., BERT on GLUE) would determine whether the phenomenon is tied to convolutional inductive biases or is a general property of overparameterized networks trained with SGD.

Practical Applications and Downstream Use Cases

Cost-efficient neural architecture search initialization. Neural architecture search (NAS) methods typically evaluate candidate architectures by training them from scratch, which is expensive. If winning tickets encode architectures that are inherently more trainable, a winning ticket architecture could serve as a strong initialization or search space prior for NAS. Rather than searching over all possible connectivity patterns, start from a winning ticket mask (discovered once on a related task) and search for modifications — prune further, add back connections, or perturb the mask. Because the winning ticket is already known to be trainable, nearby architectures in the search space are more likely to be trainable as well, reducing the number of failed training runs during search. The paper's demonstration that winning tickets exist across architectures and training configurations (Sections 2–4) provides the foundation; the specific NAS improvement would come from showing that the winning ticket mask provides a better search prior than a random or dense starting point.

Training data generation via iterative self-improvement. A winning ticket discovered for a task is, by definition, a sparse network that trains faster and generalizes better than its dense counterpart. In a self-improvement loop — where a model generates training data for a subsequent generation of models — using a winning ticket as the base architecture could accelerate each generation's training while maintaining or improving generation quality. The specific setup: train an initial dense model on a dataset, discover a winning ticket via iterative pruning, use that winning ticket to generate labels or augmentations for a larger unlabeled dataset, then train the next generation starting from the winning ticket architecture (with its original initializations). The paper's results suggest this could be more efficient than training each generation dense from scratch — the winning ticket's faster learning (2.5–3.5× earlier early-stopping for Conv-2/4/6, Section 3) and better generalization (0.3–3.5 percentage point test accuracy improvements) would compound across generations.

Edge deployment with trainable sparse models. Deploying neural networks on edge devices (phones, embedded systems, IoT) requires models that are small enough to fit in limited memory and efficient enough to run within power constraints. Standard practice is to train a large model in the cloud, then compress it for deployment via pruning or distillation — the edge device receives a frozen, compressed model that performs inference only. The lottery ticket hypothesis suggests an alternative: if the winning ticket is known (discovered once in the cloud), the edge device could receive the winning ticket architecture and initial weights, then perform lightweight on-device fine-tuning on user-specific data. Because the winning ticket trains faster than a dense network of the same size (Section 2, Figure 3: 38% earlier early-stopping at $P_m = 21\%$) and generalizes better, on-device adaptation would require fewer gradient updates and less energy. The paper's demonstration that winning tickets at $P_m = 10–20\%$ of original size match full-network accuracy on MNIST and CIFAR10 (Figures 4a, 5) provides the compression ratio; the improved training speed provides the adaptation efficiency. This use case is contingent on solving the discovery cost problem — the cloud-side winning ticket discovery would need to be made efficient enough (perhaps via one-shot pruning or early-training identification) to be worthwhile.

Scientific diagnostics for neural network optimization research. The lottery ticket hypothesis provides a new tool for studying why neural networks optimize effectively. By comparing the training dynamics of winning tickets against randomly reinitialized versions of the same architecture, researchers can isolate the contribution of initialization quality to optimization speed and final accuracy. If a proposed theory of neural network optimization (e.g., a claim about gradient descent converging to global minima under certain conditions) is correct, it should predict or explain why winning ticket initializations are special — what property of their weight values makes SGD converge faster or to better minima. The paper's random reinitialization experiment (Sections 2–4) is a ready-made diagnostic: any optimization theory can be tested by whether it distinguishes between the original winning ticket trajectory and the randomly reinitialized trajectory on the same architecture. The paper provides this diagnostic "for free" to the theory community — no new methods are needed, just the protocol of comparing $f(x; m \odot \theta_0)$ against $f(x; m \odot \theta'_0)$.