Estimating the Causal Effect of Early ArXiving on Paper Acceptance

🎯 Pitch

After adjusting for unobserved paper quality, early arXiving has at most a tiny effect on acceptance (under 4%) and often none at all—contradicting the widespread fear that posting preprints early boosts chances. Even more striking: this null effect holds for all groups, meaning elite institutions and highly cited authors gain no detectable advantage from early arXiving over their peers.

1. Executive Summary

This paper empirically investigates whether releasing a preprint on arXiv before a conference's review deadline—termed early arXiving—has a causal effect on paper acceptance, using five years of ICLR submission data (2018–2022) and applying a negative outcome control (NCO) framework from causal inference to address the unobserved confounder of paper quality (operationalizing citation count as a proxy unaffected by the treatment). A primary analysis, adjusting only for observed confounders such as author citation counts and institution rank via statistical matching, suggests a statistically significant ~10% increase in acceptance likelihood; however, after debiasing with the NCO—using binarized long-term citation counts to absorb quality-related confounding through a difference-in-difference estimator—the estimated effect shrinks to under 4% in seven of nine experimental settings and becomes statistically indistinguishable from zero in four of those nine. Stratifying by author subgroups reveals no evidence that early arXiving confers differential advantage to authors from highly-ranked institutions or with high citation counts, establishing that while a small acceptance effect may exist, it does not disproportionately benefit any particular researcher population under the assumptions of the analysis.

2. Context and Motivation

The paper addresses a tension that has grown increasingly acute in machine learning research: double-blind peer review, designed to shield reviewers from knowledge of authors' identities, coexists uneasily with the widespread practice of posting preprints to arXiv before submission deadlines. When a paper appears on arXiv prior to review, any reviewer can trivially discover the authors' names, institutions, and prior work—effectively breaking the double-blind guarantee for that submission. The central question is whether this de facto de-anonymization translates into a causal effect on acceptance decisions, and if so, whether that effect differs across author populations.

This is not a hypothetical concern. The Association for Computational Linguistics (ACL) instituted an anonymity period—a moratorium on posting accepted papers to arXiv during the review window—motivated explicitly by fears that early arXiving would advantage well-known authors and prestigious institutions at the expense of early-career researchers and those from less visible labs (Eisner et al., 2017). The policy was operational at ACL venues for years, requiring authors to refrain from publicizing their work between the submission and decision dates. Yet this policy was built on intuition and anecdote, not causal evidence. As the authors note in their epilogue, ACL abruptly removed the anonymity period policy on January 12, 2024—the same day this paper was accepted—highlighting just how live and consequential this debate remains within the research community.

The paper therefore sets out to answer two specific research questions (Section 1):

(RQ1) Does early arXiving result in different effects on authors grouped by citation counts and institution rankings? That is, does it confer an unfair advantage to already-privileged groups?
(RQ2) What effect, if any, does early arXiving have on paper acceptance overall, after accounting for the ways in which early-arXived papers systematically differ from those that are not?

Why This Problem Matters: Beyond Abstract Fairness Concerns

The practical stakes extend well beyond academic debate about reviewing philosophy. If early arXiving genuinely increases acceptance chances by a large margin—particularly for well-known authors—then the anonymity period serves a legitimate fairness function, protecting authors who either cannot or choose not to arXiv early from a systematic disadvantage. If, on the other hand, early arXiving has no meaningful causal effect after accounting for paper quality, then anonymity periods impose an unnecessary restriction on authors' freedom to disseminate their work, potentially slowing the pace of scientific communication for no tangible fairness benefit.

There is also a subtler concern that the authors do not state explicitly but which animates their analysis: reviewers may be unconsciously biased by author identity even when they believe themselves to be objective. Prior work on single- versus double-blind review provides mixed evidence on this point. Tomkins et al. (2017) found that single-blind reviewing at WSDM 2017 favored authors from highly-ranked institutions, while Madden and DeWitt (2006) found no such effect at SIGMOD. These conflicting findings suggest that any author-identity effect may be venue-specific, domain-specific, or contingent on factors the prior studies did not measure. Crucially, no prior work studied the specific mechanism of early arXiving—as distinct from single-blind review in general—on acceptance outcomes. This paper fills that gap by focusing narrowly on the causal effect of posting a preprint before the review deadline, using a dataset where all acceptance decisions (both accept and reject) are publicly observable.

The availability of ICLR's full submission and decision data is what makes this analysis possible at all. As the authors note (Section 1), ICLR is unique among major ML conferences in releasing the outcomes of all submitted papers, not just accepted ones. For any causal analysis of acceptance, having both treated and control outcomes is essential; without rejected-paper data, one cannot estimate what would have happened to early-arXived papers had they not been arXived. This data availability constraint explains why the study is restricted to a single venue and why replication at other conferences is non-trivial.

Prior Approaches and Their Shortcomings

Association studies without causal identification. The most obvious prior approach is to compute the correlation between early arXiving and acceptance—essentially, compare acceptance rates between arXived and non-arXived papers and call the difference an "effect." The paper's primary analysis (Section 5.2) does exactly this on the matched sample and finds a ~10 percentage-point advantage for early-arXived papers. But the authors immediately flag this as likely confounded: early-arXived papers in the matched sample were cited 2.13 times more in a three-year window than their non-arXived matches, strongly suggesting they differ in unmeasured quality. A naive association study conflates the effect of arXiving with the effect of being the kind of paper that authors choose to arXiv early—which is plausibly correlated with quality, author confidence, and other acceptance-relevant factors.

Controlling only for observed confounders. A more sophisticated approach—and the paper's own starting point—is to adjust for measurable confounders like author citation counts, institution rankings, paper length, topic, and so on. The authors implement this via statistical matching (Section 4.1), creating a control group of non-arXived papers that are comparable to the arXived group on all 18 observed covariates in Table 1. This eliminates overt compositional differences (e.g., arXived papers tend to be slightly longer and have more references, as shown in the pre-matching standardized mean differences in Table A.1). However, this approach still leaves the unobserved confounders—creativity, novelty, clarity of exposition, potential for impact—completely uncontrolled. The paper's causal graph (Figure 1) explicitly encodes these unobserved confounders $U$ as affecting both the treatment $A$ (early arXiving) and the outcome $Y$ (acceptance). No amount of matching on observed covariates can address bias from $U$ , and the paper's primary analysis (red bars in Figure 2) therefore represents a confounded estimate, as the authors acknowledge.

Prior work on citation effects of preprints. Feldman et al. (2018) observed an association between early arXiving and increased citation counts in the calendar year of publication. However, as the authors point out (Section 4.2), this work explicitly disclaims causal interpretation and does not account for unobserved quality confounders. Moreover, Feldman et al. measured citations by calendar year, which introduces a mechanical bias: early-arXived papers have been publicly available for more of that calendar year than papers first appearing in conference proceedings, inflating their citation counts independent of any quality difference. The present paper addresses this by counting citations from each paper's individual first-public-appearance date, creating a fair temporal comparison. The concerns about "flag-planting"—authors using arXiv to claim priority and garner citations that would otherwise go to later-appearing papers—are acknowledged (Goldberg, 2017) but the authors note the absence of empirical evidence for this mechanism and express skepticism that it would be widespread enough to invalidate citation count as a negative control.

Absence of causal frameworks in peer-review research. Perhaps the most significant gap the paper identifies is methodological: prior work on peer review fairness has largely operated in an associational paradigm, computing correlations between author characteristics and outcomes without a formal causal identification strategy. The NLP+ML community has become increasingly sophisticated about causal inference in recent years (see Feder et al., 2022 for a survey), but this sophistication had not been brought to bear on the specific question of early arXiving effects. The paper positions itself as introducing a principled causal framework—complete with a causal graph, explicit identification assumptions, and a negative control outcome design—to a question that had previously been debated on the basis of intuition, anecdote, and confounded correlations.

How This Paper Positions Itself

The paper positions itself at the intersection of two methodological traditions: the potential outcomes framework (Rubin, 1974, 2005) for defining and estimating causal effects, and the negative control outcome (NCO) method (Rosenbaum, 1989a; Lipsitch et al., 2010) for addressing unobserved confounding. The authors do not claim to invent either technique; rather, they argue that the peer-review setting presents a natural application where an NCO—paper citation count—is unusually well-motivated, and where the combination of matching (for observed confounders) plus NCO-based debiasing (for unobserved confounders) yields more credible estimates than either approach alone.

The paper's key methodological move is to operationalize paper quality through citation count, binarized at various thresholds, and to treat this as an NCO: a variable affected by the same unobserved confounders as acceptance but not causally affected by early arXiving. The validity of this choice rests on the argument that the number of citations a paper receives in a fixed time window after its first public appearance is a function of its quality, novelty, and impact—the very unobserved confounders that plague the acceptance analysis—but is not caused by whether the paper happened to be posted on arXiv before the review deadline. The authors defend this assumption by (a) using each paper's individual first-public-appearance date as the citation-counting start point, eliminating the calendar-year bias in prior work; (b) acknowledging and rebutting the flag-planting concern; and (c) conducting sensitivity analyses across multiple citation-count windows ( $n = 1, 2, 3$ years) and multiple binarization thresholds ( $q = 50\%, 75\%, 90\%$ quantiles), showing that the qualitative finding—NCO adjustment substantially reduces the estimated effect—is robust.

The paper explicitly avoids claiming to have proven the absence of an early-arXiving effect. The language throughout is carefully conditional: estimates are valid "under standard assumptions" and "under a range of reasonable assumptions." The authors acknowledge that the NCO assumption itself is "uncheckable" (Limitations section), and they end by calling for a randomized controlled trial as the gold standard. This epistemic humility is not just rhetorical caution—it reflects the genuine limitation of any observational causal study, where identification always rests on assumptions that cannot be empirically verified from the data at hand.

Relative to the prior associational literature, the paper's contribution is not a single definitive answer but rather a shift in the burden of proof. Before this work, the default assumption in policy discussions seemed to be that early arXiving probably helps well-known authors and that anonymity periods are therefore justified as a precautionary measure. The paper's analysis suggests that once quality differences are accounted for—even imperfectly, through the NCO—the evidence for a large or differential effect largely evaporates. This does not disprove the existence of an effect, but it means that advocates of anonymity periods can no longer point to a clear empirical basis for their position; the ball is in the court of those claiming a causal effect to demonstrate it, ideally through a randomized trial.

A final layer of positioning concerns the paper's relationship to the ACL policy change that coincided with its acceptance. The epilogue is unusually personal for a technical paper, noting that the authors' curiosity about "whether the policy's original goals and motivations were held in practice" directly motivated the work, and acknowledging that "we are unlikely to know whether this paper contributed to this policy lift from ACL." This transparency about the real-world policy stakes distinguishes the paper from purely methodological exercises and frames it as a contribution to an ongoing, consequential debate within the research community.

3. Technical Approach

3.1 Reader Orientation

This is an observational causal inference study that builds a statistical pipeline to estimate the causal effect of a binary treatment (whether a paper was posted to arXiv before the ICLR review deadline) on a binary outcome (whether the paper was accepted), using five years of real-world conference submission data where no randomized experiment was conducted. The core idea is that standard association analyses conflate the effect of early arXiving with the effect of being the kind of high-quality paper that authors choose to arXiv early, so the paper applies a negative control outcome (NCO) framework—which uses a paper's long-term citation count as a proxy that absorbs unmeasured quality confounding—combined with statistical matching on observed covariates to isolate the treatment effect from both measured and unmeasured confounders.

The problem the system solves is: given observational data where treatment assignment is non-random and correlated with unobservable quality, how do we estimate what would have happened to early-arXived papers had they NOT been arXived? The solution's "shape" is a two-stage pipeline: first, create a matched control group of non-arXived papers that are statistically indistinguishable from the arXived group on 18 observed covariates (matching); second, apply a difference-in-difference estimator between the acceptance outcome and a binarized citation-count NCO to cancel out the biasing effect of unobserved confounders (NCO debiasing). The final output is the average treatment effect on the treated (ATET): the estimated change in acceptance probability caused specifically by early arXiving, for the subpopulation of papers that were actually arXived early.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a pipeline:

Data ingestion and feature extraction — takes raw ICLR submission data (paper PDFs, metadata, OpenReview profiles) from 2018–2022 and constructs 18 observed covariates per submission (paper content features, author gender indicators, institution rankings, citation counts) plus the treatment assignment (early arXived vs. not) and the outcome (accepted vs. rejected). Also queries the Semantic Scholar API to obtain each paper's exact first-public-appearance date and its n-year citation counts.
Statistical matching — takes the treated group (1,486 early-arXived papers) and the untreated pool (7,493 non-arXived papers), and for each treated paper finds a matched control paper using tripartite matching on all 18 observed covariates. The matching enforces near-exact balance on categorical variables (year, number of authors), minimizes L2 distance on numerical covariates, and achieves fine-balance on topic cluster distributions. The output is 1,486 matched pairs where the only systematic difference between groups is the treatment.
Negative control outcome (NCO) construction — takes each paper's citation count in a fixed n-year window (n = 1, 2, or 3) from its individual first-public-appearance date, and binarizes it at a quantile threshold q (50%, 75%, or 90%) to produce a binary variable N(n)_q indicating whether the paper is "highly cited." The NCO is designed to share confounders (especially quality) with the acceptance outcome but not be causally affected by early arXiving.
Difference-in-difference (DiD) estimator — takes the matched pairs and the NCO, and computes the ATET as the difference (across treatment groups) of the difference between the acceptance outcome Y and the NCO N. This cancels out additive confounding from unobserved variables under the assumption that U affects Y and N identically.
Stratified subgroup analysis — applies the same DiD+ NCO pipeline separately within author subgroups defined by institution rank (top-10, top-11-to-100, others) and citation count (<500, 500-2000, >2000), producing stratum-specific ATET estimates with bootstrap confidence intervals to test for differential effects across groups.

Information flows sequentially: raw data → covariate extraction → matching → NCO construction → DiD estimation → subgroup stratification. Each step depends on the output of the previous step, and the final ATET estimates are computed on the matched sample with NCO debiasing.

3.3 Roadmap for the Deep Dive

First, the potential outcomes framework and estimand definition — because everything downstream depends on understanding what "causal effect" means in this setting and why the ATET (rather than the ATE) is the target.
Second, the identification assumptions — because the validity of every subsequent estimate rests on these (unverifiable) conditions, and understanding which assumptions do what work is essential for interpreting the results.
Third, the matching procedure — because matching is the first stage that creates the analytic sample, and its design choices (tripartite matching, fine-balance, covariate selection) determine what observed confounding is controlled for.
Fourth, the negative control outcome framework and difference-in-difference estimator — because this is the paper's core methodological contribution for addressing unobserved confounding, and the mechanics of how DiD uses the NCO to debias the ATET require careful step-by-step explanation.
Fifth, the NCO variable construction and validity argument — because the entire NCO approach hinges on citation count satisfying the negative control condition, and the paper's defense of this choice (date-based citation windows, binarization, robustness across thresholds) needs detailed scrutiny.
Sixth, the estimation procedure — because the specific computational steps (logistic outcome models, bootstrap confidence intervals, combining matching with DiD) determine how the reported numbers in Figures 2 and 3 are produced.

This ordering follows the natural dependency chain: we need to define what we're estimating before we can explain how we estimate it, and we need to explain the assumptions before we can evaluate whether the estimator is credible under those assumptions.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an applied causal inference paper whose core idea is that a negative control outcome—paper citation count, binarized at various thresholds—can absorb the confounding effect of unmeasured paper quality on the relationship between early arXiving and acceptance, and that combining this NCO with careful statistical matching on observed covariates yields a more credible estimate of the causal effect than either matching alone or naive association.

Potential Outcomes Framework and the ATET Estimand

The paper operates within the Rubin causal model (Rubin, 1974), which conceptualizes causal effects as contrasts between potential outcomes—what would happen to each unit under different treatment conditions, only one of which is ever observed. The treatment variable $A \in \{0,1\}$ is binary: $A=1$ if the paper was posted to arXiv before the ICLR reviewing deadline (early arXiving), and $A=0$ otherwise. The outcome variable $Y \in \{0,1\}$ is also binary: $Y=1$ if the paper was accepted to ICLR, and $Y=0$ if rejected.

For each submitted paper, two potential outcomes exist:

$Y_{A=1}, Y_{A=0}$

where $Y_{A=1}$ is the acceptance outcome that would occur if the paper were early-arXived, and $Y_{A=0}$ is the acceptance outcome that would occur if the same paper were not early-arXived.

What these expressions encode: For a single paper, $Y_{A=1}$ and $Y_{A=0}$ represent the two possible worlds—one where the authors chose to arXiv early and one where they did not. Only one of these worlds is realized: either the paper was actually arXived early (so we observe $Y_{A=1}$ and $Y_{A=0}$ is counterfactual), or it was not (so we observe $Y_{A=0}$ and $Y_{A=1}$ is counterfactual). The fundamental challenge is that the counterfactual can never be observed; it must be estimated from the data of other papers that received the other treatment, under assumptions that make such borrowing valid.

Why this formulation: The potential outcomes framework forces explicitness about what is being estimated and what assumptions license that estimation. It distinguishes the definition of the causal effect (a contrast between potential outcomes) from the estimation problem (how to impute the unobserved counterfactual), which is essential because different identification strategies—matching, NCO, randomized trials—are different solutions to the same underlying missing-data problem.

The paper's target estimand is the average treatment effect on the treated (ATET):

$\text{ATET} = \mathbb{E}[Y_{A=1} - Y_{A=0} \mid A = 1]$

where $\mathbb{E}[\cdot]$ is the expectation (average) over the population of early-arXived papers, $Y_{A=1}$ is the potential outcome under treatment (early arXiving), $Y_{A=0}$ is the potential outcome under no treatment for the same papers, and the conditioning $| A = 1$ restricts the average to only those papers that were actually early-arXived.

What it computes: The ATET takes the subpopulation of papers that were in fact arXived early, and for each such paper asks: what was the difference between the outcome that occurred (acceptance status under early arXiving) and the outcome that would have occurred had the same paper not been arXived? It then averages these individual-level differences. Operationally, this answers: "For the group of authors who chose to arXiv early, did that choice help them relative to the counterfactual where they had not arXived?"

Why ATET rather than ATE (average treatment effect on the entire population): The ATE would average over all papers—those that were arXived early and those that were not—asking what would happen if we forced every paper in the population to be arXived vs. not. But the policy question motivating this work is specifically about the papers that do get arXived early: are those papers gaining an advantage? The ATET is the natural estimand for this question because it acknowledges that treatment uptake is not random—authors self-select into arXiving—and focuses on the effect for those who actually receive the treatment. Estimating the ATE would require imputing what would happen if papers whose authors chose not to arXiv were forced to do so, which is a more speculative counterfactual given that these papers likely differ from arXived papers in unobserved ways related to why their authors made that choice.

The ATET ranges from $-100\%$ to $+100\%$ , where positive values indicate that early arXiving increases acceptance probability, negative values indicate it decreases acceptance probability, and values near zero indicate no meaningful causal effect.

Identification Assumptions

Because only one potential outcome is observed for each paper, the ATET cannot be computed directly from data—it requires identifying assumptions that license the use of observed outcomes from untreated papers as valid proxies for the counterfactual outcomes of treated papers. The paper states four standard assumptions (Assumption 3.1), plus one additional assumption specific to the NCO framework:

1. Ignorability (no unmeasured confounding, given covariates):

$\{Y_{A=0}, Y_{A=1}\} \perp\!\!\!\perp A \mid (C, U)$

where $Y_{A=0}$ and $Y_{A=1}$ are the potential outcomes, $A$ is the treatment assignment, $C$ represents the observed confounders (the 18 variables in Table 1), and $U$ represents the unobserved confounders (paper quality, novelty, etc.).

What this means: Conditional on both observed covariates $C$ and unobserved covariates $U$ , the treatment assignment is independent of the potential outcomes. In plainer language: once we account for everything that affects both the decision to arXiv early and the chances of acceptance, whether a paper happened to be arXived or not is as good as random with respect to its acceptance prospects. There are no other hidden variables creating a spurious correlation between treatment and outcome.

The vertical bar notation $\perp\!\!\!\perp A \mid (C,U)$ means that within any subgroup defined by specific values of $C$ and $U$ , the distribution of potential outcomes is the same for arXived and non-arXived papers. This is the assumption that makes it valid to use non-arXived papers as a control group for arXived papers, provided we condition on enough covariates.

Why this form: Ignorability is the foundational assumption of all observational causal inference. Without it, any observed difference between treated and control groups could be due to unmeasured confounding rather than a causal effect. The paper's key methodological challenge is that $U$ cannot be measured directly, which is why the NCO framework is needed—to relax the requirement that we condition on $U$ explicitly by instead using a proxy variable that shares $U$ 's confounding structure.

2. Positivity (overlap in covariate distributions):

$0 < P(A = 1 \mid C = c, U = u) < 1$

What this means: For any combination of observed and unobserved covariates $(c,u)$ that exists in the population, there must be some papers with that covariate profile that were arXived early and some that were not. The probability of receiving treatment cannot be exactly 0 or exactly 1 anywhere in the covariate space.

Why this constraint matters: If there exists a subgroup of papers for which arXiving is impossible (e.g., all papers on a certain topic always arXiv early), then we have no control papers to serve as counterfactuals for those treated papers, and the ATET cannot be estimated for that subgroup without extrapolating from other covariate regions—a model-dependent exercise that relies on untestable assumptions about how effects vary across covariate space.

3. Consistency (potential outcomes align with observed outcomes):

$Y_a = Y^{\text{obs}} \quad \text{if} \quad A^{\text{obs}} = a$

where $Y_a$ is the potential outcome under treatment level $a$ , $Y^{\text{obs}}$ is the observed outcome, and $A^{\text{obs}}$ is the observed treatment assignment.

What this means: For a paper that was actually arXived early ( $A^{\text{obs}} = 1$ ), the potential outcome $Y_{A=1}$ equals whatever acceptance outcome we observe. For a paper that was not arXived ( $A^{\text{obs}} = 0$ ), $Y_{A=0}$ equals the observed outcome. This assumption links the mathematical potential-outcome notation to the actual data.

Why this is an assumption rather than a definition: Consistency rules out situations where the treatment has different "versions" that produce different outcomes. For example, if "early arXiving" could mean posting a complete polished draft vs. posting a rough placeholder, and these versions have different effects on acceptance, the simple binary treatment $A$ might not capture the relevant variation, and the observed outcome for an arXived paper might not equal the potential outcome under the specific version of treatment we intend to study. The paper implicitly assumes that any variation within "early arXiving" does not affect outcomes differently enough to matter.

4. Negative control condition (the NCO is not causally affected by treatment):

$N_a = N \quad \text{for} \quad a = 0, 1$

where $N$ is the negative control outcome (citation count), and $N_a$ is the potential outcome for the NCO under treatment level $a$ .

What this means: The value of the NCO does not depend on whether the paper was early-arXived. For a given paper, its long-term citation count would be the same regardless of whether the authors chose to arXiv early or not. This means the NCO is not a mediator—it is not on the causal pathway from treatment to outcome.

Why this assumption is essential for the NCO framework: If the treatment does affect the NCO, then the NCO contains not only the confounding signal we want to cancel out but also a treatment-effect signal. Subtracting the NCO from the outcome (as the DiD estimator does) would then subtract part of the true treatment effect, biasing the estimate downward. The entire validity of the NCO approach rests on this assumption being approximately true.

5. Additive equi-confounding (Assumption 4.1, specific to DiD):

$\mathbb{E}[Y_a(1) - Y_a(0) \mid U, A=a, C] = \mathbb{E}[Y_a(1) - Y_a(0) \mid A=a, C]$

where $Y_a(1)$ and $Y_a(0)$ are potential outcomes at two "time periods"—in the NCO analogy, the acceptance outcome $Y$ and the NCO $N$ respectively, and the conditioning on $(U, A=a, C)$ vs. only $(A=a, C)$ indicates that the expected difference between $Y$ and $N$ does not depend on $U$ .

What this means in operational terms: The unobserved confounders $U$ affect the outcome $Y$ and the NCO $N$ in the same way, so that the expected difference $Y - N$ is constant across different levels of $U$ . This is the assumption that lets us "difference out" the confounding: by computing $(Y - N)$ for the treated group and $(Y - N)$ for the control group, then taking the difference of these differences, any additive effect of $U$ cancels out because it appears identically in both $Y$ and $N$ .

Why this specific form: The additive equi-confounding assumption is weaker than assuming $U$ has no effect (which would be false—paper quality clearly affects both acceptance and citations), but stronger than allowing arbitrary effects of $U$ on $Y$ and $N$ . It assumes that $U$ shifts $Y$ and $N$ by the same amount, which is made more plausible by the paper's binarization of the citation count: transforming the raw citation count into a binary "highly-cited" variable puts it on approximately the same scale as the binary acceptance outcome, making the additive assumption more defensible than it would be for a continuous citation count that could be shifted by hundreds of citations while acceptance shifts by at most 1 (0 to 1).

Together, these five assumptions jointly identify the ATET from the observed data: ignorability (with $U$ handled via NCO) allows treated and control groups to be compared; positivity ensures such comparison is possible; consistency links potential outcomes to data; the negative control condition prevents the NCO from absorbing the treatment effect; and additive equi-confounding makes the DiD correction valid.

Statistical Matching Procedure

Before applying the NCO-based estimator, the paper creates a matched control group to balance the observed confounders $C$ between treated and control papers. The matching procedure takes the 1,486 early-arXived papers (the treated group) and the pool of 7,493 non-early-arXived papers (the untreated pool), and produces 1,486 matched pairs such that the treated and control groups are statistically indistinguishable on all 18 covariates listed in Table 1.

The 18 observed confounders (Table 1): The paper uses an extensive set of covariates spanning paper content, author characteristics, and institution attributes:

Paper content features (7 variables): year (conference year, categorical with levels 2018–2022), n_fig (number of figures, integer), n_ref (number of references, integer), n_sec (number of sections, integer), log_text_length (logarithm of token count, float), text_ppl (text perplexity scored by RoBERTa, float in (0,1) where higher values indicate more fluent text), and topic_cluster (one of 20 topic categories assigned by spectral clustering on SPECTER embeddings of paper abstracts, categorical).

Author and institution features (11 variables): n_author (total number of authors, integer), n_author_female (number of female-identified authors, integer), first_author_female (binary indicator for whether first author identifies as female), any_author_female (binary indicator for any female authors), no_US_author (binary indicator for whether no authors are US-based), log_inst_rank_min (log of the lowest-ranked—i.e., worst—institution among paper authors, float), log_inst_rank_avg (log of average institution rank, float), log_inst_rank_max (log of the highest-ranked—i.e., best—institution, float), log_author_cite_min (log of citation count of least-cited author, float), log_author_cite_avg (log of average author citation count, float), and log_author_cite_max (log of citation count of most-cited author, float).

Matching algorithm: Tripartite matching. The paper uses the tripartite matching algorithm of Zhang et al. (2021), which extends standard propensity-score or distance-based matching to handle three distinct matching objectives simultaneously:

Numerical variable matching: For continuous and count covariates (n_fig, n_ref, n_sec, log_text_length, text_ppl, and all logged institution-rank and citation-count variables), the algorithm minimizes the L2 (Euclidean) distance between treated and matched control units, penalizing large discrepancies. This ensures that the matched pairs are similar on average across all these dimensions without requiring exact equality.
Near-exact matching on categorical variables: The variables n_author (total author count) and year (conference year) are nearly exactly matched, meaning the algorithm strongly prioritizes pairing papers from the same year with the same number of authors. The qualifier "nearly" indicates that if an exact match is impossible for some treated unit, a small deviation is allowed rather than dropping that unit entirely, but exact matches are preferred whenever feasible in the data.
Fine-balance on topic clusters: The distribution of the topic_cluster variable (a 20-level categorical variable) in the matched control group is constrained to be identical to its distribution in the treated group. This means that if, say, 7.3% of early-arXived papers are in the "DL/NLP/Transformer/LM" cluster, then exactly 7.3% of the matched control papers must also be in that cluster. Fine-balance is weaker than exact pair-level matching on topic (it allows individual pairs to have different topics as long as the marginal distributions match), but it prevents the matched sample from being compositionally different from the treated group on this important covariate. The acronym "DL/NLP/Transformer/LM" stands for Deep Learning / Natural Language Processing / Transformer / Language Model.

Why tripartite matching rather than propensity score matching: Standard propensity score matching (Rosenbaum and Rubin, 1983) collapses all covariates into a single scalar (the estimated probability of treatment given covariates) and matches on that scalar. While this is asymptotically valid—matching on the propensity score balances all covariates in expectation—it can perform poorly in finite samples, especially with many covariates, because it reduces a multidimensional balancing problem to a one-dimensional one, potentially leaving some covariates imbalanced. Tripartite matching addresses this by explicitly optimizing three different objectives (numerical distance, exact categorical matching, fine-balance) simultaneously, giving direct control over the balance achieved on each type of covariate. The paper notes that after matching, all standardized mean differences (SMDs) between treated and matched control groups are substantially reduced, with topics achieving fine-balance (Table A.1).

Matching results (Table A.1): The paper reports standardized mean differences before and after matching for all covariates in Table A.1. Before matching, the largest SMDs are for log_text_length (SMD = -0.355, indicating arXived papers tend to be longer), n_ref (SMD = -0.224), n_sec (SMD = -0.213), and n_fig (SMD = -0.221). After matching, all SMDs are below 0.04 in absolute value, and most are below 0.02, indicating excellent covariate balance. The topic cluster distribution is exactly balanced (fine-balance achieved) with an SMD effectively zero.

The matching yields a final analytic sample of 1,486 matched pairs, meaning all 1,486 treated papers were successfully matched to a control paper. No treated papers were discarded due to inability to find a match, which is important because it means the ATET estimate applies to the full population of early-arXived papers rather than only to a matchable subset.

A critical design choice: The matching is performed before applying the NCO correction, and all subsequent analyses are conducted on the matched sample. This is methodologically important because it means the NCO is used only to address residual confounding from unobserved variables $U$ , not to compensate for imbalance in observed covariates $C$ that could have been handled through matching. This "doubly robust" flavor—matching for observed confounders, NCO for unobserved confounders—makes the analysis more credible than either approach alone would be, because each method addresses a distinct source of bias and the failure of one does not necessarily invalidate the other.

Negative Control Outcome Framework and Difference-in-Difference Estimator

Even after matching, the treated and control groups may differ on unobserved confounders $U$ such as paper quality, novelty, and potential impact. The NCO framework addresses this by introducing a proxy variable $N$ that is affected by the same confounders as the outcome $Y$ but is not causally affected by the treatment $A$ . The intuition is: if we observe that early-arXived papers have higher acceptance rates and higher NCO values than matched controls, the elevation in acceptance may be due to quality differences ( $U$ ) rather than a causal effect of $A$ . By subtracting the NCO's signal from the outcome's signal, we can isolate the part of the acceptance difference that is not explained by quality.

The DiD estimator in the NCO context. The paper follows Sofer et al. (2016) in establishing an equivalence between the NCO framework and the difference-in-difference (DiD) estimator from longitudinal studies. In a standard DiD setting, outcomes are measured at two time periods ( $t=0$ and $t=1$ ), and the treatment occurs between them. The ATET is:

$\text{ATET} = \mathbb{E}[Y_1(1) - Y_0(1)] - \mathbb{E}[Y_1(0) - Y_0(0)]$

where $Y_1(1)$ is the treated group's outcome after treatment (post-period, treated), $Y_0(1)$ is the control group's outcome after treatment (post-period, control), $Y_1(0)$ is the treated group's outcome before treatment (pre-period, treated), and $Y_0(0)$ is the control group's outcome before treatment (pre-period, control).

What this computes: The DiD estimator first computes the change over time for the treated group: $\mathbb{E}[Y_1(1) - Y_1(0)]$ , which combines the true treatment effect plus any time trend. It then computes the change over time for the control group: $\mathbb{E}[Y_0(1) - Y_0(0)]$ , which captures only the time trend (since the control group was not treated). Subtracting the control group's change from the treated group's change cancels out the shared time trend, leaving only the treatment effect.

Why this structure applies to the NCO setting: Sofer et al. (2016) note that the role of the pre-period outcome $Y(0)$ in a DiD study is analogous to the NCO $N$ : both are variables that share confounders with the post-treatment outcome but are unaffected by the treatment. By substituting $Y(0) \rightarrow N$ , the DiD formula becomes:

$\text{ATET} = \mathbb{E}[Y_1 - N_1] - \mathbb{E}[Y_0 - N_0]$

where $Y_1$ is the acceptance outcome for early-arXived (treated) papers, $N_1$ is the NCO for those same papers, $Y_0$ is the acceptance outcome for non-arXived (control) papers, and $N_0$ is the NCO for the control papers.

Operationalizing the NCO DiD: To estimate the ATET, the paper:

Computes the difference $(Y - N)$ for each paper in the matched sample—that is, acceptance status minus the binary NCO.
Averages $(Y - N)$ separately for the treated group ( $A=1$ ) and the control group ( $A=0$ ).
Subtracts the control-group average from the treated-group average.

The result is a single number: the estimated ATET. If the NCO absorbs all confounding from $U$ , and the additive equi-confounding assumption holds, this number represents the causal effect of early arXiving on acceptance probability.

Why the additive equi-confounding assumption is necessary: The subtraction $\mathbb{E}[Y_1 - N_1] - \mathbb{E}[Y_0 - N_0]$ only cancels out confounding if $U$ affects $Y$ and $N$ identically. If $U$ has a larger effect on $Y$ than on $N$ , then $\mathbb{E}[Y_1 - N_1]$ will still contain residual confounding that is not fully subtracted away by $\mathbb{E}[Y_0 - N_0]$ , and the ATET estimate will be biased upward. Conversely, if $U$ has a larger effect on $N$ than on $Y$ , the ATET will be biased downward (over-corrected). The binarization of the citation count—collapsing it to a 0/1 variable—is intended to make the scales of $Y$ and $N$ more comparable, strengthening the plausibility of additive equi-confounding.

Estimation details for binary outcomes: Because both $Y$ and $N$ are binary, the paper uses a logistic outcome model (implicitly, via the conditional means) to compute the expected values in the DiD formula. Bootstrap resampling is used to construct 95% confidence intervals for the ATET, accounting for the uncertainty in both the matching step and the DiD estimation. The specific procedure pools the matched pairs, resamples with replacement, recomputes the DiD estimator on each bootstrap sample, and takes the 2.5th and 97.5th percentiles of the bootstrap distribution as the confidence interval.

Negative Control Outcome Variable Construction and Validity

The paper uses a paper's citation count as the basis for the NCO. The raw variable is $CC(n)$ , the number of citations received by a paper in the $n$ -year window following its first public appearance online, for $n \in \{1, 2, 3\}$ . This raw count is then binarized at a quantile threshold $q \in \{0.5, 0.75, 0.9\}$ to produce the binary NCO:

$N^{(n)}_q = \mathbb{1}[CC(n) > CC(n)_q]$

where $CC(n)$ is the paper's citation count in the $n$ -year window after its first public appearance, $CC(n)_q$ is the $q$ -th quantile of the citation count distribution in the relevant (matched) sample, and $\mathbb{1}[\cdot]$ is the indicator function returning 1 if the condition is true and 0 otherwise. When $q = 0.5$ , the threshold is the median citation count; when $q = 0.9$ , the threshold is the 90th percentile.

What this computes: $N^{(n)}_q = 1$ means the paper is in the top $(1-q)$ fraction of citation counts—i.e., it is "highly cited" relative to its cohort. $N^{(n)}_q = 0$ means it is not highly cited. The nine combinations of $n \in \{1,2,3\}$ and $q \in \{0.5, 0.75, 0.9\}$ produce nine distinct NCO variables, with different implied definitions of what counts as "highly cited."

Why binarize rather than use raw citation counts: The paper gives two motivations. First, the additive equi-confounding assumption requires $Y$ and $N$ to be on comparable scales, and binarizing the citation count puts it on the same 0/1 scale as the acceptance outcome. Raw citation counts vary over orders of magnitude (from 0 to hundreds), and assuming that $U$ shifts raw citation counts and acceptance probabilities by the same additive amount would be implausible. Second, the binarization "smooths out" minor fluctuations in citation counts—a paper cited 10 times vs. 11 times is likely not meaningfully different in quality, and treating both as "not highly cited" (if below the threshold) avoids overfitting to small, noisy differences.

Why multiple thresholds and windows: By sweeping over $n \in \{1,2,3\}$ and $q \in \{0.5, 0.75, 0.9\}$ , the paper conducts a sensitivity analysis. If the findings are consistent across these choices—particularly for larger $n$ and higher $q$ , which the authors argue are the "strongest" NCO choices because long-term, high-threshold citations are least likely to be causally affected by early arXiving—then the results are robust to the specific operationalization. The nine settings in Figure 2 allow the reader to assess this directly.

The critical validity argument: why citation count satisfies the negative control condition. The negative control condition $N_a = N$ requires that early arXiving does not causally affect the NCO. The paper defends this choice against two potential objections:

Objection 1: Prior work found an association between arXiving and higher citations. Feldman et al. (2018) observed that arXived papers receive more citations in their calendar year of publication. The paper counters on two grounds: (a) that work was explicitly associational and did not control for quality confounders, so it does not demonstrate a causal effect; (b) that work measured citations by calendar year, which mechanically advantages arXived papers because they were publicly available for more of that calendar year. By using each paper's individual first-public-appearance date as the start of the citation-counting window, the present paper eliminates this mechanical bias.

Objection 2: Flag-planting—authors use arXiv to stake a priority claim and divert citations from later-appearing papers. The paper acknowledges this concern but notes the absence of empirical evidence for a widespread flag-planting effect, expresses skepticism that it would be large enough to invalidate citation count as an NCO, and explicitly invites future empirical work on the question.

The date-based citation counting mechanism: A crucial implementation detail is how $CC(n)$ is computed. For each paper, the Semantic Scholar API provides the date of first public appearance—the earliest date among all sources where the paper appeared (arXiv, conference proceedings, institutional repository, etc.). For an early-arXived paper, this is typically the arXiv posting date. For a non-arXived paper, this is the conference proceedings release date. $CC(n)$ counts citations from other papers whose own publication dates fall within $n$ years of this first-appearance date. This ensures that early-arXived papers do not get a "head start" advantage in citation counting—the n-year window starts when each paper first becomes visible, not from some common calendar reference point.

Sample size variation across NCO settings: Because $CC(n)$ requires $n$ years of post-publication citation data, later conference years are excluded from analyses using larger $n$ . Table B.1 shows that for $CC(1)$ , all 1,486 matched pairs from conference years 2018–2022 are available. For $CC(2)$ , conferences 2018–2021 are used, yielding 1,073 matched pairs. For $CC(3)$ , only conferences 2018–2020 are used, yielding 570 matched pairs. This sample attrition with larger $n$ means that the ATET estimates for different $n$ are computed on different subsets of the data, which must be considered when comparing results across panels in Figure 2.

Evidence motivating the choice of NCO (Table B.2): The paper shows that in the matched sample, early-arXived papers have both higher acceptance rates AND substantially higher citation counts. For instance, among 2018 submissions, early-arXived papers had an average $CC(3)$ of 249.95 citations vs. 47.40 for matched controls—a 5.3× difference. This large citation gap in a matched sample (where observed covariates are balanced) strongly suggests the presence of unobserved quality confounders that affect both acceptance and citations, which is precisely the scenario where an NCO can help.

Estimation Procedure and Inference

The paper estimates the ATET under each of the nine NCO specifications as follows:

Step 1: Matching on observed covariates. Using the full dataset of 10,297 ICLR submissions, identify the 1,486 early-arXived papers and match each to a non-arXived control using tripartite matching, producing the matched sample. This step is performed once and the matched pairs are fixed for all subsequent analyses.

Step 2: Subset to the relevant conference years. For a given $n$ (say, $n=3$ ), restrict the matched sample to conference years where $CC(n)$ can be computed (2018–2020 for $n=3$ ). This yields 570 matched pairs for $n=3$ (Table B.1).

Step 3: Compute raw citation counts. For each paper in the restricted matched sample, query the Semantic Scholar API to obtain $CC(n)$ , the number of citations received within $n$ years of the paper's first public appearance.

Step 4: Binarize at quantile $q$ . Compute the $q$ -th quantile $CC(n)_q$ of the $CC(n)$ distribution in the restricted matched sample. Define $N^{(n)}_q = 1$ if $CC(n) > CC(n)_q$ , and 0 otherwise.

Step 5: Fit a logistic outcome model (implicitly). The paper uses a logistic regression model to relate $Y$ (acceptance) to $A$ (treatment), $N$ (the NCO), and the interaction between $A$ and $N$ , or equivalently computes the conditional expectations $\mathbb{E}[Y \mid A=1]$ and $\mathbb{E}[Y \mid A=0]$ and $\mathbb{E}[N \mid A=1]$ and $\mathbb{E}[N \mid A=0]$ directly from the matched sample. The DiD estimator uses these expectations to compute the ATET.

Step 6: Compute the DiD estimate. Calculate:

$\widehat{\text{ATET}} = (\bar{Y}_1 - \bar{N}_1) - (\bar{Y}_0 - \bar{N}_0)$

where $\bar{Y}_1$ is the mean acceptance rate among treated papers in the matched sample, $\bar{N}_1$ is the mean NCO among treated papers, $\bar{Y}_0$ is the mean acceptance rate among control papers, and $\bar{N}_0$ is the mean NCO among control papers. For a binary NCO, this is equivalent to taking the difference in acceptance rates, then subtracting the difference in "highly-cited" rates.

Step 7: Bootstrap for confidence intervals. The entire procedure—including the matching step—is bootstrapped to produce 95% confidence intervals. The paper resamples the matched pairs with replacement, recomputes the DiD estimator on each bootstrap sample, and takes the empirical 2.5th and 97.5th percentiles of the bootstrap distribution. Bootstrapping the matching step (rather than treating the matched sample as fixed) appropriately accounts for the uncertainty introduced by the matching procedure itself.

Subgroup analyses (Figure 3): The same DiD+ NCO procedure is applied separately within subgroups defined by (a) institution rank, grouped as "top-10," "top-10 to 100," and "others" based on the minimum-ranked institution among the paper's authors, and (b) author citation count, grouped as "<500," "500-2000," and ">2000" based on the maximum citation count among the paper's authors. The stratification variables (institution rank and citation count) are removed from the covariate set when performing matching within that subgroup to avoid conditioning on the stratification variable. The paper acknowledges that some strata have small sample sizes (Table B.3 and B.4), leading to wider confidence intervals and reduced statistical power to detect differential effects.

The "Unadj" (unadjusted) baseline in Figures 2 and 3: For comparison, the paper also reports the ATET estimated without using the NCO—that is, simply the difference in mean acceptance rates between treated and control groups in the matched sample, without any DiD correction. This "Unadj" estimate (shown in red in Figure 2) represents the best estimate achievable through matching alone, and the difference between the Unadj estimate and the NCO-adjusted estimate quantifies how much of the apparent treatment effect is attributable to unobserved confounders captured by the NCO.

Why the Unadj baseline values differ across panels in Figure 2: Because the sample changes with $n$ (later conference years are dropped for larger $n$ ), the "Unadj" estimate is recomputed on each subset to ensure a fair comparison with the NCO-adjusted estimate on the same data. The Unadj values of 9.79%, 9.90%, and 10.03% correspond to the 1-year, 2-year, and 3-year citation-count subsets respectively.

Alternative: Quantile-Quantile Equi-Confounding (Appendix C.1)

The paper also explores a weaker alternative to the additive equi-confounding assumption called quantile-quantile (QQ) equi-confounding (Sofer et al., 2016; related to the change-in-change method of Athey and Imbens, 2006). This alternative relaxes two restrictions of the additive assumption:

The QQ approach allows $Y$ and $N$ to be affected by different sets of unobserved confounders ( $U$ for $Y$ and $W$ for $N$ ), rather than requiring the same $U$ to affect both.
The QQ approach is invariant to monotone transformations of $N$ , meaning the raw (un-binarized) citation count can be used without worrying about scale differences between $Y$ and $N$ .

The QQ estimator works by matching the quantiles of the distributions rather than the means. Define the QQ association between the unobserved confounder and the treatment as:

$q_0(u \mid c) = F_{U \mid A=0, C=c} \circ F^{-1}_{U \mid A=1, C=c}(u)$

$q_1(w \mid c) = F_{W \mid A=0, C=c} \circ F^{-1}_{W \mid A=1, C=c}(w)$

where $F_{X \mid Z}$ is the cumulative distribution function (CDF) of $X$ given $Z$ , $F^{-1}_{X \mid Z}$ is the inverse CDF (quantile function), $u$ and $w$ are quantile levels in $[0,1]$ , $U$ is the unobserved confounder affecting $Y$ , $W$ is the unobserved confounder affecting $N$ , $C$ is observed covariates, and $\circ$ denotes function composition (applying the inverse CDF then the CDF, which maps quantiles of the treated distribution to quantiles of the control distribution).

What the QQ equi-confounding assumption (Assumption C.1) states:

$q_0(v \mid c) = q_1(v \mid c) \quad \text{for} \quad v \in [0,1]$

This means that the relationship between $U$ and treatment assignment—expressed on the quantile scale—is identical to the relationship between $W$ and treatment assignment. In plainer language: the way the unobserved confounders are distributed across treatment groups follows the same pattern for both $Y$ and $N$ , even though the confounders themselves may be different variables and the outcomes may be on completely different scales.

Why this is weaker than additive equi-confounding: The additive assumption requires $U$ to affect $Y$ and $N$ by the same additive amount, which forces $Y$ and $N$ to be on the same scale and to share the same confounding structure. The QQ assumption only requires the confounding structure to match after transforming to quantiles, which is automatically satisfied if both confounders have the same distributional relationship with treatment, regardless of scale or magnitude.

Theorem 1 (ATET under QQ equi-confounding): Under the QQ assumption, the ATET can be expressed as:

$\text{ATET} = \mathbb{E}[Y \mid A = 1] - \mathbb{E}[\tilde{Y}]$

where $\tilde{Y} = F^{-1}_{Y \mid A=0, C} \circ F_{N \mid A=0, C}(N^*)$ , and $N^* \sim (N \mid A=1, C)$ is a random variable distributed as the NCO in the treated group. The term $F^{-1}_{Y \mid A=0, C}(F_{N \mid A=0, C}(N^*))$ maps a treated-group NCO value to its quantile in the control-group NCO distribution, then maps that quantile to the corresponding value in the control-group outcome distribution—essentially answering: "if a treated paper had this NCO value, what acceptance outcome would we predict for a control paper with the same NCO quantile?"

Results under QQ equi-confounding: Using raw $CC(3)$ as the NCO (without binarization), the paper estimates:

$\widehat{\text{ATET}} = -4.375\%$

with a 95% bootstrap confidence interval of $(-9.965\%, -0.092\%)$ .

The estimate is negative and the confidence interval barely excludes zero, suggesting a weak negative effect or no effect. The paper notes that while the direction differs from the additive DiD results (which showed small positive effects), the significance levels are consistent: in both analyses, there is no strong evidence for a large or robust effect of early arXiving on acceptance.

Why the QQ analysis is relegated to an appendix: The QQ approach is more technically complex and the paper's main text prioritizes the additive DiD approach because (a) the binarized NCO makes the additive assumption more defensible, (b) the DiD estimator is more familiar and interpretable, and (c) the results are qualitatively similar (effects are small and often insignificant). The QQ analysis serves as a robustness check confirming that the main findings are not purely artifacts of the additive assumption or the binarization procedure.

4. Key Insights and Innovations

Innovation 1: The NCO Framework Transforms an Unverifiable Debate into a Falsifiable Empirical Question

The paper's deepest contribution is not any specific numerical estimate but rather the epistemological reframing it brings to the early-arXiving debate. Before this work, the discussion about whether posting preprints advantages certain authors operated largely in the realm of intuition, anecdote, and associational evidence. Proponents of anonymity periods argued that de-anonymization must help well-known authors (Eisner et al., 2017); skeptics pointed to the absence of clear empirical evidence for bias in double-blind settings (Madden and DeWitt, 2006); and the resulting stalemate produced policies—like the ACL anonymity period—based on precautionary reasoning rather than empirical justification.

What makes this paper's contribution fundamental rather than incremental is that it converts an unverifiable claim into a structured causal inference problem with explicit, testable assumptions. The causal graph (Figure 1) makes the structure of the problem visible: it says, "here are the variables at play, here is where the unmeasured confounders live, and here is what we would need to believe for an estimate to be credible." This may sound like standard causal inference practice, but in the specific context of peer-review fairness research—where prior work had largely operated in an associational paradigm (comparing means between groups, computing correlations between author attributes and outcomes)—the introduction of a formal identification strategy marks a qualitative shift in what counts as evidence.

Consider the contrast with the primary analysis alone. If the paper had stopped after matching and reported a ~10% acceptance advantage for early-arXived papers, it would have added one more associational data point to a literature already full of conflicting findings. The result would have been cited by both sides—supporters of anonymity periods as evidence of bias, skeptics as merely confounded by quality—and the debate would have continued unchanged. The paper's key move is to say: let's design a variable that can absorb the quality confound we all agree exists, and see what happens to the estimate. The NCO framework operationalizes the skeptic's objection ("early-arXived papers are just better") and incorporates it directly into the estimation procedure.

This is methodologically novel in the peer-review literature but conceptually significant beyond the specific domain. The paper demonstrates a template for using naturally-occurring proxy variables to address unmeasured confounding in studies of scientific evaluation—a class of problems that includes peer review, grant funding, hiring, and promotion. The specific move of using citation count as an NCO works because of a confluence of properties unique to the scientific domain: citations are affected by the same quality confounders as acceptance, they are measured post-treatment (allowing the DiD analogy), and their relationship to the treatment can be argued away with careful design choices (date-based windows, binarization). This template could generalize to other evaluation settings where post-decision outcomes (patent citations, commercial impact, downstream publications) capture the unobserved merit that reviewers are supposed to be identifying.

The paper's epistemic humility—acknowledging that the NCO assumption is "uncheckable" and calling for a randomized trial—is itself part of the innovation. By making the assumptions transparent and testable through sensitivity analyses (the grid of nine NCO specifications in Figure 2), the paper shifts the burden of proof: rather than asking skeptics to prove the absence of an effect, it asks proponents of an effect to explain why the NCO correction is insufficient. This is a more productive framing for an inherently unverifiable observational question.

Innovation 2: Quality Confounding Is Empirically Demonstrated, Not Just Hypothesized, Through the Citation Gap

A common rhetorical move in observational studies is to invoke unmeasured confounding as a caveat—"of course, quality differences might explain this, but we can't measure quality so we'll assume it away." The paper does something more powerful: it empirically demonstrates the magnitude of quality confounding by revealing a dramatic citation gap between treated and matched control papers, even after exhaustive matching on observed covariates.

The evidence is in Table B.2. For 2018 submissions in the matched sample—where early-arXived and control papers are balanced on all 18 covariates including author citation counts, institution rankings, paper length, topic, and so on—the average three-year citation count for arXived papers is 249.95, compared to 47.40 for matched controls. This is a 5.3× difference that cannot be attributed to any measured confounder (they're balanced) or to the direct effect of arXiving on citations (the NCO assumption says this path doesn't exist). The only remaining explanation is that the matching succeeded in making the groups look similar on the variables we can measure, but failed to make them similar on the variables we cannot measure—and those unmeasured variables (call them quality, novelty, impact, or clarity) are powerful enough to produce a factor-of-5 difference in downstream citations.

This finding is significant not because it's surprising—most researchers would guess that better papers get arXived early and also get cited more—but because it quantifies the selection bias in a way that prior work did not. The primary analysis's ~10% acceptance advantage looks plausible as a causal effect only if we believe that after controlling for 18 covariates, the remaining quality difference between arXived and non-arXived papers is modest enough to produce at most a small boost in outcomes. The citation data says otherwise: the residual quality difference is enormous, easily large enough to explain a 10-percentage-point acceptance gap without any causal effect of arXiving at all. The NCO correction reduces the estimated ATET from ~10% to under 4% in most settings, and the citation gap provides the empirical justification for why this reduction makes sense—there was a lot of quality confounding to absorb.

This is a diagnostic move that could be exported to other causal inference studies: use a post-treatment proxy variable not just in the estimator, but as a descriptive tool to show the reader how much selection bias likely remains after conditioning on observables. The paper does not need to claim that citation count is a perfect measure of quality—only that the observed citation gap is too large to be consistent with a scenario where residual confounding is negligible. This turns the standard caveat ("unmeasured confounding might exist") into a substantive finding ("unmeasured confounding is large enough to explain the entire apparent treatment effect, and here's the evidence").

Innovation 3: The Absence of Differential Subgroup Effects Is Itself an Informative Finding

The paper's subgroup analysis (Figure 3) finds that early arXiving does not confer a statistically distinguishable advantage to authors from highly-ranked institutions or with high citation counts, compared to authors from lower-ranked institutions or with fewer citations. The confidence intervals across subgroups overlap substantially across all nine NCO specifications, under both the additive DiD approach and the QQ equi-confounding alternative. This null finding—no evidence of differential effects—is the paper's answer to RQ1 and is, in context, more consequential than the overall ATET estimate.

To understand why, consider the policy debate that motivates the work. The ACL anonymity period (Eisner et al., 2017) was instituted not because early arXiving was thought to uniformly boost acceptance rates for all papers, but because it was thought to specifically advantage "authors from prestigious institutions, or who are well-known in the community" (Section 5.3). The fairness concern was distributional: even if the average effect were zero, an anonymity-breaking mechanism that helped the already-privileged and hurt the less-privileged would be objectionable on equity grounds. The paper tests this distributional claim directly and finds no supporting evidence.

This is not a failure to reject the null due to insufficient statistical power—though the authors acknowledge that some strata have small sample sizes (Tables B.3 and B.4), leading to wider confidence intervals. It is rather that the point estimates themselves show no consistent pattern of advantage for high-ranked groups. In some specifications, the estimated ATET for top-10 institution authors is actually lower than for authors from other institutions (e.g., in Figure 3(a), the 50%-Qtl row shows ATETs of -9.11% for top-10 vs. -3.33% for "others"). If a systematic bias in favor of elite institutions existed, we would expect the point estimates to consistently tilt in that direction, even if individual estimates were noisy. The observed pattern is essentially flat across groups, suggesting that the noise is not masking a consistent directional effect.

The significance of this finding goes beyond the specific debate about anonymity periods. It suggests that reviewers may be better at compartmentalizing author identity than the policy discourse assumes. The concern animating double-blind review is that knowing an author's identity triggers cognitive biases—halo effects around prestigious institutions, leniency toward familiar names, discounting of unknown researchers—that reviewers cannot suppress even when they try to evaluate the work on its merits. If this were true in a strong form, we would expect it to manifest in the subgroup analysis as a positive ATET for high-status authors (arXiving helps them by revealing their identity) and a negative or zero ATET for low-status authors (arXiving either doesn't help or actively hurts by revealing their lack of status). The absence of this pattern does not prove that bias doesn't exist—reviewers might be biased in ways that are uncorrelated with the institution and citation measures used here—but it undercuts the most straightforward mechanism by which early arXiving was hypothesized to create unfairness.

The paper is appropriately cautious about this interpretation. The confidence intervals are wide, the sample sizes in some strata are small, and the NCO framework's assumptions apply within each subgroup separately. But as a piece of evidence in a policy debate that had previously been conducted largely on the basis of first principles, the finding that the data show no differential effect—across multiple operationalizations of "status," multiple NCO specifications, and multiple citation-count windows—carries weight that a single statistically significant point estimate would not.

Innovation 4: The Convergence of Additive DiD and Quantile-Quantile Methods Under Different Assumptions Strengthens the Overall Conclusion

A recurring challenge in observational causal inference is that different estimation strategies—each valid under different untestable assumptions—can produce different answers, leaving the analyst uncertain which to trust. The paper addresses this by implementing two NCO-based estimators that rest on qualitatively different identification assumptions—the additive equi-confounding assumption for the main DiD analysis and the quantile-quantile equi-confounding assumption for the appendix analysis—and showing that both point toward the same substantive conclusion: the effect of early arXiving on acceptance is small and uncertain.

The two sets of assumptions are not minor variations on a theme; they make different demands of the data and are vulnerable to different failure modes. The additive DiD approach (main text) requires $Y$ and $N$ to share both the same confounding structure and the same scale, which motivates the binarization of citation counts. If this assumption fails because the unobserved confounders affect citations and acceptance on different scales—say, a quality shock that doubles citations but only increases acceptance probability by 10 percentage points—the DiD estimator will over-correct or under-correct. The QQ approach (Appendix C.1) relaxes both requirements: it allows $Y$ and $N$ to be affected by different sets of confounders and to be on completely different scales, requiring only that the distributional relationship between confounders and treatment is the same for both outcomes. Its failure mode is different: if the shape of the confounding distribution differs between $Y$ and $N$ (e.g., quality affects citations with a long right tail but affects acceptance more uniformly), the QQ estimator will be biased in ways the additive estimator might not be.

The fact that both approaches yield qualitatively consistent results—the additive DiD produces small positive or near-zero ATETs in most specifications; the QQ approach produces an ATET of -4.375% with a confidence interval of (-9.97%, -0.09%) that barely excludes zero—is a form of triangulation across assumptions. Neither approach is "correct" in an absolute sense (both rely on unverifiable conditions), but the convergence of their conclusions under different assumption sets makes the overall finding more robust than either estimate alone would be. If the additive approach had produced a large positive ATET while the QQ approach produced zero, we would rightly be suspicious that the additive result was an artifact of the binarization or the equal-scale assumption. The observed consistency suggests instead that the signal in the data genuinely points toward a small effect, and that the assumptions needed to extract that signal are not driving the result.

Beyond the specific application, this analysis pattern—implementing estimators under multiple incomparable identification assumptions and looking for convergence in conclusions—represents a useful methodological template for applied causal inference in settings where no single set of assumptions is clearly correct. It echoes the call in the causal inference methodology literature for sensitivity analyses that probe the robustness of findings to alternative identifying assumptions (Rosenbaum, 1989a; Lipsitch et al., 2010), but implements it concretely in a way that produces interpretable, side-by-side results. For NLP researchers applying causal methods to observational text data (the audience for Feder et al., 2022), the paper provides a worked example of how to do this in practice without requiring advanced causal inference expertise.

Innovation 5: The Study's Negative Result, Combined with ACL's Policy Reversal, Creates a Natural Experiment for the Role of Empirical Evidence in Policy-Making

This final innovation is meta-scientific rather than technical: the paper, by virtue of its timing and its explicit connection to the ACL anonymity period policy, provides a case study in how—and whether—empirical causal evidence influences institutional policy decisions in the research community. The authors note in their epilogue that on January 12, 2024, the same day the paper was accepted at CLeaR, the ACL removed its anonymity period policy with immediate effect, and they acknowledge that "we are unlikely to know whether this paper contributed to this policy lift."

The paper was first made available as a preprint on June 24, 2023—roughly six months before the ACL policy change. Whether or not it directly influenced the ACL's decision, the paper's findings are now part of the public record on a question that had been debated for years without systematic causal evidence. The paper does not claim to have settled the debate—it explicitly calls for a randomized controlled trial—but it does shift the evidentiary landscape: before June 2023, the best available evidence on early-arXiving effects was essentially anecdotal; after June 2023, there exists a peer-reviewed, methodologically-sophisticated analysis finding that the effects, if they exist at all, are small, non-differential across author groups, and substantially confounded by paper quality.

This is not a "contribution" in the traditional sense of advancing a technical method or discovering a new empirical regularity. It is rather a demonstration of what happens when a previously assumption-driven policy domain encounters structured causal evidence. The ACL's policy change creates a natural experiment of sorts: researchers studying scientometrics and peer review can now observe, over subsequent conference cycles, whether the removal of the anonymity period coincides with changes in acceptance patterns across author subgroups. If the paper's null finding is correct—if early arXiving truly does not create differential advantage—then the removal of the policy should not produce observable shifts in the demographic composition of accepted papers. If such shifts do occur, they would constitute evidence against the paper's conclusions, potentially motivating a re-examination of the NCO assumptions or the discovery of bias mechanisms not captured by the institution-rank and citation-count stratifications used here.

The paper's unusual epilogue—which would be out of place in a purely methodological contribution—reflects the authors' recognition that their work sits at a rare intersection where a technical analysis can have immediate, observable policy consequences. The epistemic modesty of the final sentences—"As with many real-world causal questions, assessing the true causal effect is hard"—is not just a rhetorical flourish; it acknowledges that even a well-executed observational study cannot close the book on a causal question, and that the ultimate test will come from observing what happens when the policy changes in the real world. This framing elevates the paper from a one-off empirical exercise to an ongoing contribution to a living policy debate, with the paper's methods and findings serving as a baseline against which future evidence—observational and experimental—can be compared.

5. Experimental Analysis

Evaluation Methodology

Dataset. The study uses the ICLR 2018–2022 submission database assembled by Zhang et al. (2022), containing 10,297 submitted papers with their acceptance decisions (3,678 accepted, 6,619 rejected). From this, 1,486 papers were identified as early-arXived (the treated group) and matched to 1,486 non-early-arXived controls, yielding an analytic sample of up to 2,972 papers. When citation-count NCOs are used, later conference years are dropped: CC(1) retains all 1,486 matched pairs (years 2018–2022), CC(2) retains 1,073 pairs (years 2018–2021), and CC(3) retains 570 pairs (years 2018–2020) per Table B.1.
Data sources and matching to external databases. Each ICLR submission is matched to a unique document in the Semantic Scholar (S2) Academic Graph (Kinney et al., 2023) via fuzzy title matching; S2 provides each paper's first-public-appearance date and n-year citation counts. The matching success rate is 99% for accepted papers and 81% for rejected papers (Table A.2); of the matched papers, roughly 90% have publication dates available. Papers without publication dates are discarded (6% of early-arXived, 11% of non-arXived), and rejected papers that could not be matched to S2 are assigned 0 citations.
Metrics. The primary outcome is binary acceptance to ICLR (accept vs. reject). The target estimand is the average treatment effect on the treated (ATET), expressed as a percentage point difference in acceptance probability: ATET = E[Y_{A=1} - Y_{A=0} | A = 1]. The ATET ranges from -100% to +100%, with positive values indicating that early arXiving increases acceptance chances. Confidence intervals are computed at the 95% level via bootstrap resampling.
Baselines. The paper uses two primary reference points. First, the "Unadj" (unadjusted) estimate — the simple difference in mean acceptance rates between treated and matched control groups in the analytic sample, without any NCO correction. This estimates the treatment effect after matching on observed covariates but without addressing unobserved quality confounding, and it differs across citation-count subsets because later years are excluded for larger n (Figure 2, red bars: 9.79% for the CC(1) subset, 9.90% for CC(2), 10.03% for CC(3)). Second, the pre-matching (unmatched) primary analysis — the crude association between early arXiving and acceptance in the full dataset, before matching or NCO correction, which the paper reports as significant and substantial but does not tabulate as a formal baseline in the figures (the matched-sample Unadj serves as the primary comparison).
Analysis samples and budget accounting. The paper does not use a computational budget in the FLOPs or GPU-hours sense; the "cost" of the analysis is measured in sample attrition. As the NCO uses longer citation-count windows (n = 2 or 3 years), more recent conference years must be excluded because their n-year citation windows have not yet elapsed, reducing the matched-pair count from 1,486 to 1,073 to 570. The paper conducts analyses across all three sample sizes to assess robustness to this attrition, and the "Unadj" baseline is always recomputed on the same subset as the NCO-adjusted estimate to maintain comparability.
Cross-validation and statistical protocol. The paper does not use cross-validation in the predictive modeling sense; the statistical protocol consists of (1) one-shot tripartite matching to create the matched pairs, (2) DiD estimation on the matched sample under each of the nine NCO specifications, and (3) bootstrap resampling of the matched pairs (resampling the pairs with replacement, recomputing the DiD estimator on each bootstrap replicate) to construct 95% confidence intervals. The bootstrap accounts for uncertainty in both the matching and the estimation steps because it resamples the pairs rather than treating the matched sample as fixed. For the subgroup analyses (Figure 3), the same protocol is applied separately within each stratum defined by institution rank or author citation count.

Main Quantitative Results

Primary Analysis: Effect of Early ArXiving After Matching Only (No NCO Correction)

The paper's starting point is the estimated ATET after statistical matching on all 18 observed covariates but without any correction for unobserved confounders. These are the "Unadj" estimates shown in red in Figure 2:

On the CC(1) subset (n = 1,486 matched pairs, conference years 2018–2022): ATET = 9.79% . This is the acceptance-rate difference between early-arXived papers and their matched controls.
On the CC(2) subset (n = 1,073 pairs, years 2018–2021): ATET = 9.90% .
On the CC(3) subset (n = 570 pairs, years 2018–2020): ATET = 10.03% .

The three estimates are tightly clustered around 10% despite being computed on different subsets of the data, suggesting the association is stable across conference years. All three are statistically significant at the 95% level (confidence intervals do not contain zero, visible in Figure 2). Interpreted at face value, these results suggest that early-arXived papers are approximately 10 percentage points more likely to be accepted than observationally similar papers that were not arXived early—a substantial effect given that overall ICLR acceptance rates during this period ranged from roughly 25–35%.

However, the paper immediately contextualizes these numbers with the citation-count evidence from Table B.2. In the matched sample for 2018, early-arXived papers had an average CC(3) of 249.95 citations compared to 47.40 for matched controls — a 5.3× difference. This citation gap exists despite the matched pairs being balanced on all 18 covariates, implying that the two groups differ substantially on unobserved dimensions (plausibly paper quality) that affect both citations and acceptance. The "Unadj" estimate of ~10% therefore likely confounds any actual causal effect of early arXiving with the effect of being a higher-quality paper.

NCO-Adjusted Analysis: Effect of Early ArXiving After Debiasing with Citation-Based NCOs

The central results of the paper are the nine NCO-adjusted ATET estimates shown in the non-red bars of Figure 2, each corresponding to a different combination of citation-count window n (1, 2, or 3 years) and binarization quantile q (50%, 75%, or 90%):

One-year citation count (top panel of Figure 2):

N^{(1)}_{0.5} (median threshold): ATET = -0.66%, 95% CI crosses zero → not significant at the 95% level.
N^{(1)}_{0.75}: ATET = 3.73%, 95% CI excludes zero → significant .
N^{(1)}_{0.90}: ATET = 7.56%, 95% CI excludes zero → significant .

Two-year citation count (middle panel of Figure 2):

N^{(2)}_{0.5}: ATET = -4.06%, 95% CI crosses zero → not significant .
N^{(2)}_{0.75}: ATET = 0.74%, 95% CI crosses zero → not significant .
N^{(2)}_{0.90}: ATET = 5.13%, 95% CI excludes zero → significant .

Three-year citation count (bottom panel of Figure 2):

N^{(3)}_{0.5}: ATET = -9.17%, 95% CI crosses zero → not significant (though the point estimate is large in magnitude, the wide confidence interval on the smaller n = 570 sample renders it insignificant).
N^{(3)}_{0.75}: ATET = -2.63%, 95% CI crosses zero → not significant .
N^{(3)}_{0.90}: ATET = 2.16%, 95% CI crosses zero → not significant .

Pattern across NCO specifications. Two clear trends emerge from Figure 2:

NCO adjustment substantially reduces the estimated effect. In all nine specifications, the NCO-adjusted ATET is smaller in magnitude than the corresponding "Unadj" estimate (the red bar in the same panel). The reduction is dramatic: from ~10% without NCO correction to values that are either statistically insignificant or under ~5% in most specifications. This is the core empirical evidence that quality confounding explains a large portion of the apparent early-arXiving advantage.
Longer citation-count windows and higher thresholds produce weaker effects. The authors argue that N^{(n)}q with larger n and higher q are the "strongest" NCO choices because they capture whether a paper is highly-cited over a longer period and are thus least likely to be causally affected by early arXiving. Under N^{(3)}{0.75} (ATET = -2.63%, ns) and N^{(3)}{0.90} (ATET = 2.16%, ns)—the two specifications the authors consider most credible—the estimated effect is small in magnitude and statistically indistinguishable from zero. Under N^{(1)}{0.90} (ATET = 7.56%, significant) and N^{(2)}_{0.90} (ATET = 5.13%, significant), some significant effects persist, but these use shorter citation-count windows where the NCO assumption is arguably weaker.

The paper's summary claim. The authors state that early arXiving increases acceptance likelihood by less than 4% in seven out of nine settings and the effect is not statistically significant in four out of nine settings (Section 6, restated in the executive summary). These counts are directly verifiable from Figure 2: the seven settings with ATET < 4% are N^{(1)}{0.5} (-0.66%), N^{(1)}{0.75} (3.73%), N^{(2)}{0.5} (-4.06%), N^{(2)}{0.75} (0.74%), N^{(3)}{0.5} (-9.17% — note the absolute value < 4%? Actually -9.17% exceeds 4% in magnitude; the claim "less than 4%" appears to mean an absolute increase less than 4 percentage points, so -9.17% would qualify as an effect smaller than 4% in the positive direction. The paper's phrasing "increasing acceptance likelihood by less than 4%" implies a positive upper bound, which does not strictly apply to the negative point estimates. The intended meaning is that the effect size is small and below 4 percentage points in most settings); the four insignificant settings are N^{(1)}{0.5}, N^{(2)}{0.5}, N^{(3)}{0.75}, and N^{(3)}_{0.90}.

Subgroup Analysis: Differential Effects by Institution Rank and Author Citation Count

Figure 3 presents ATET estimates stratified by two author-group variables (institution rank and citation count), under each of the nine NCO specifications. The headline finding for RQ1 is that no statistically significant differences in the treatment effect are observed across any subgroups: confidence intervals overlap across all strata in all panels.

Stratification by minimum institution rank (Figures 3(a), 3(c), 3(e)):

Authors are grouped into "Top-10" institutions, "Top-10 to 100" institutions, and "Others" based on the minimum (worst) institutional rank among the paper's co-authors.
Under the Unadj baseline (no NCO correction), ATET point estimates vary across groups but with overlapping confidence intervals. For example, using the CC(3) sample (Figure 3(a), top row): ATET = 2.37% for Top-10, 12.11% for Top-10-to-100, and 12.17% for Others. The Top-10 estimate is substantially lower (contrary to the hypothesis that elite institutions benefit most), but the wide confidence intervals (especially for Top-10, where sample sizes are small per Table B.3) make these differences non-significant.
After NCO adjustment with N^{(3)}_{0.90} (Figure 3(a), bottom row): ATET = -4.92% for Top-10, 7.64% for Top-10-to-100, and 11.80% for Others. Again, the Top-10 point estimate is actually negative while Others is positive, but the confidence intervals are wide and overlap.
Across all nine NCO specifications and all three institution-rank strata, the pattern is inconsistent—sometimes Top-10 shows the highest ATET, sometimes the lowest—and no configuration shows a clear, statistically significant advantage for higher-ranked institutions.

Stratification by maximum author citation count (Figures 3(b), 3(d), 3(f)):

Authors are grouped into "< 500 citations," "500-2000 citations," and "> 2000 citations" based on the maximum citation count among the paper's co-authors.
The same pattern of overlapping confidence intervals and inconsistent point-estimate ordering holds. For instance, under N^{(3)}_{0.90} (Figure 3(b), bottom row): ATET = 2.03% for all authors, -4.62% for < 500 citations, 2.16% for 500-2000, and 2.42% for > 2000. The lowest-citation group actually shows a more negative point estimate than the highest-citation group, which is directionally opposite to the hypothesized bias, but the differences are not statistically significant.
Sample sizes in some strata are small (Table B.4): for the CC(3) sample at the 90% quantile, the "500-2000 citations" treated group contains only 6 papers when A = 0, resulting in very wide confidence intervals that limit the power to detect differential effects.

Key takeaway from Figure 3. Even in the Unadj analysis (no NCO correction), where the overall ATET is ~10% and significant, there is no evidence that this effect differs across author subgroups. The NCO-adjusted analyses—which shrink the overall effect toward zero—similarly show no subgroup heterogeneity. This finding directly addresses the core fairness concern motivating anonymity periods: if early arXiving conferred a differential advantage to high-status authors, we would expect subgroup-specific ATETs to be consistently higher for top-ranked institutions and highly-cited authors. The data do not support this pattern under any specification.

Alternative QQ Equi-Confounding Analysis (Appendix C.1)

The paper supplements the main DiD analysis with an alternative NCO estimator based on quantile-quantile (QQ) equi-confounding, which uses raw (un-binarized) CC(3) as the NCO and relaxes the additive equi-confounding assumption. The key result:

ATET = -4.375%, 95% bootstrap CI = (-9.965%, -0.092%) .

The confidence interval barely excludes zero (the upper bound is -0.092%), indicating the effect is at most very weakly negative. The paper notes that while the direction differs from the additive DiD analysis (which mostly produces positive point estimates in the significant cases), the significance level is consistent: both approaches find no strong evidence for a substantial positive effect, and the effect estimates are small in absolute magnitude relative to the Unadj baseline.

The QQ-plot (Figure C.1) shows clear departure from the identity line, which encodes the presence of unobserved confounding (Sofer et al., 2016). This departure provides visual confirmation that the NCO framework is capturing real confounding structure, not merely adding noise that shrinks estimates through attenuation bias.

Ablation Studies and Robustness Checks

Choice of citation-count window (n): The paper sweeps n ∈ {1, 2, 3} across all nine NCO specifications in Figure 2. The estimated ATET decreases as n increases (moving from the top panel to the bottom panel), which is consistent with the authors' argument that longer-term citation counts are better NCOs because they are less likely to be causally affected by early arXiving. The number of matched pairs drops from 1,486 to 1,073 to 570 (Table B.1), so the wider confidence intervals at n = 3 partly reflect reduced sample size, but the point estimates themselves also systematically shrink.

Choice of binarization threshold (q): The paper sweeps q ∈ {0.5, 0.75, 0.9} within each panel of Figure 2. Higher thresholds (stricter definitions of "highly cited") tend to produce larger ATET point estimates within a given n. For example, within the CC(3) panel: ATET = -9.17% at q = 0.5, -2.63% at q = 0.75, and 2.16% at q = 0.9. The authors argue that higher-q NCOs are "stronger" because extremely highly-cited papers (top 10%) are even less likely to have their citation counts causally affected by early arXiving than moderately-cited papers (top 50%). The pattern of ATET estimates across q is not monotonic in the direction one would expect if the NCO were simply absorbing random noise (which would shrink all estimates toward zero equally); instead, the effects remain distinct and follow a consistent pattern across n.

Matching quality (Table A.1): The paper reports standardized mean differences (SMDs) before and after matching for all 18 covariates. Before matching, notable imbalances exist: log_text_length (SMD = -0.355, arXived papers longer), n_ref (SMD = -0.224), n_sec (SMD = -0.213), and text_ppl (SMD = 0.172). After tripartite matching, all SMDs are below 0.04 in absolute value, with most below 0.02, indicating excellent covariate balance. The topic_cluster distribution is exactly balanced across the 20 topics (fine-balance achieved, SMD < 0.001). This evidence supports the claim that the matched sample is well-balanced on observables, meaning residual ATET differences are unlikely to be driven by the covariates listed in Table 1.

Inclusion of text-based confounders: The paper includes text_ppl (a document-level fluency measure computed from RoBERTa) and log_text_length as observed confounders, which is atypical for peer-review studies that often focus exclusively on author and institution metadata. The pre-matching SMDs confirm that these text features differ between arXived and non-arXived papers (arXived papers are longer and have slightly different fluency scores), justifying their inclusion. However, the paper does not report an ablation showing ATET estimates without text features, leaving unclear how much these variables contribute to reducing residual confounding relative to author-metadata features alone.

Stratification by author subgroups with varying sample sizes (Tables B.3, B.4): The stratified analyses replicate the main DiD + NCO procedure within subgroups, but sample sizes become small in some cells. For the CC(3) subset at the 90% quantile, the "Top-10" institution subgroup contains only 4 treated and 2 control papers (Table B.3), and the "< 500 citations" subgroup contains 2 treated and 2 control papers (Table B.4). The authors explicitly acknowledge that "some strata have few samples, making it hard for statistical procedures to be sensible, which renders a considerably wider confidence interval" (Appendix B). The stratum-level ATET estimates in Figures 3(a) and 3(b) show wide confidence intervals for the smallest subgroups, but the overall conclusion of "no differential effect" is drawn from the full set of estimates across all nine NCO specifications, where the pattern of overlapping CIs is consistent.

Impact of excluding papers without S2 matches or publication dates: Table A.2 shows that 19% of rejected papers could not be matched to S2 (likely never published anywhere), and roughly 10% of matched papers lacked publication dates. The unmatched rejected papers are assigned 0 citations. If these papers are systematically lower-quality than matched rejected papers—which seems plausible, since papers never published anywhere are likely weaker—then assigning them 0 citations could affect the quantile thresholds used to binarize CC(n), potentially making the "highly-cited" threshold easier to clear. The paper does not report a sensitivity analysis that excludes papers with missing S2 matches entirely.

Robustness to the specific choice of NCO variable: The paper's NCO is citation count, but it briefly considers (and rejects) alternative operationalizations. The QQ analysis (Appendix C.1) uses raw citation counts without binarization, serving as a sensitivity check on the binarization step. The paper does not test alternative NCOs (e.g., citation count from a different database, download counts, social media attention), which would further probe the robustness of the NCO assumption.

Negative result: ReST^{EM}-style optimization was not attempted or needed. Unlike the companion reference paper (which found that RL-based revision model optimization degraded performance), the current paper has no analogous "failed experiment" because its analysis is purely observational—there is no model training or optimization loop that could fail. The closest analog is the acknowledgment that the NCO assumption itself is "uncheckable" (Limitations section) and that the positive findings for N^{(1)}{0.90} and N^{(2)}{0.90} (which remain significant after adjustment) could reflect either a true causal effect or incomplete confounding control.

Critical Assessment

Does the evidence support the claim that early arXiving has at most a small effect on acceptance (RQ2)? The paper's primary claim is that after NCO adjustment, the effect is under 4% in seven of nine settings and insignificant in four of nine. The numbers in Figure 2 directly support these counts. However, the interpretation of these counts as evidence for a "small effect" requires accepting the NCO assumptions, particularly the additive equi-confounding assumption. If this assumption fails—for instance, if quality affects citations more strongly than it affects acceptance—the NCO correction could over-adjust, turning a genuine moderate positive effect into a small or negative estimate. The QQ analysis (Appendix C.1) provides some reassurance by producing a consistent qualitative conclusion under a different set of assumptions, but it also produces a negative point estimate (-4.375%), which is harder to interpret as "a small positive effect that is being over-corrected" versus "genuinely no effect or a slight negative effect."

A more precise statement of what the experiments demonstrate: the apparent ~10% acceptance advantage of early-arXived papers is substantially confounded by quality, and after attempting to correct for this using citation-count NCOs under plausible assumptions, the residual effect is small and often statistically insignificant. The experiments do not rule out a genuine effect of 3-5 percentage points—several adjusted estimates fall in this range (N^{(1)}{0.75} = 3.73%, N^{(2)}{0.90} = 5.13%) and are statistically significant—but they establish that any such effect is much smaller than the naive association suggests and is not robustly detectable across all reasonable NCO specifications.

Does the evidence support the claim of no differential effect across author subgroups (RQ1)? The subgroup analyses in Figure 3 consistently show overlapping confidence intervals across strata, which is the standard criterion for "no significant difference." However, the statistical power to detect differential effects is limited by the small sample sizes in some strata (particularly for CC(3) at high quantiles, where some subgroups contain fewer than 10 papers). The absence of evidence for differential effects is not evidence of absence, especially when the subgroup ATET estimates themselves are mostly insignificant (so there is little "signal" to differ across groups in the first place). The strongest evidence for the no-differential-effect claim comes from the consistency of the overlapping-CI pattern across all nine NCO specifications and both stratification variables: it is not a single underpowered test but a replicated null finding.

Weaknesses and missing analyses:

Single venue, single time period. All data come from ICLR 2018–2022. ICLR's review process, acceptance rates, and reviewer pool differ from other major ML conferences (NeurIPS, ICML, ACL, etc.), and the arXiving culture may vary across subcommunities. The paper's findings may not generalize to other venues, and the authors acknowledge this limitation explicitly.

The NCO assumption is untestable. The claim that citation count is not causally affected by early arXiving is central to the entire analysis and cannot be verified from the available data. The authors defend it by argumentation (date-based citation windows, binarization, longer windows being less susceptible) but do not provide empirical evidence that the effect of early arXiving on citations is zero. If flag-planting or increased visibility from arXiving does affect citation counts even in multi-year windows, the NCO adjustment is biased. The sensitivity analysis across n and q addresses this indirectly (if the NCO assumption were severely violated, we might not see the systematic shrinkage pattern across specifications), but does not directly test the assumption.

Difficulty estimation cost is irrelevant here, but data attrition is. The companion reference paper's compute-optimal framework required expensive difficulty estimation; the current paper's "cost" is data attrition—larger n values exclude more recent conference years, reducing the matched sample from 1,486 to 570. The most credible NCO specifications (n = 3) have the smallest sample sizes and thus the widest confidence intervals, making it harder to detect a true effect if one exists. The tradeoff between NCO credibility and statistical power is unavoidable with the available data but limits the precision of the study's central estimates.

No direct quality measurement. The paper treats quality as unobserved and uses citation count as a proxy, but never attempts to construct a direct quality measure (e.g., expert ratings, post-hoc impact scores). This is a deliberate design choice—if a good quality measure existed, the NCO framework would be unnecessary—but it means the paper cannot validate the claim that citation count is actually capturing quality rather than something else (e.g., author network effects, topic popularity, self-promotion).

The Unadj baseline is a moving target. The Unadj estimate changes slightly across the three citation-count subsets (9.79% to 10.03%), meaning the "headline" confounded effect depends on which subset of conference years is analyzed. This variation is small relative to the NCO adjustment magnitude, but it means the claim "the unadjusted effect is ~10%" actually refers to three different numbers computed on three different samples, making the comparison with adjusted estimates slightly less clean than it appears.

No comparison with alternative debiasing methods. The paper uses DiD + NCO as its sole approach to addressing unobserved confounding (with the QQ method as a robustness check). Alternative approaches like instrumental variables (if a variable affecting arXiving propensity but not acceptance could be found), regression discontinuity (if there were a sharp cutoff in arXiving behavior), or sensitivity analyses that bound the ATET under varying assumptions about the strength of unobserved confounding (e.g., Rosenbaum bounds) are not explored. The paper's conclusion that the effect is "small" depends on the specific assumptions of the NCO framework; alternative frameworks might produce different bounds.

The arXiv-to-arXiving mapping. The paper defines treatment as posting to arXiv before the reviewing deadline, but does not verify that all papers posted to arXiv actually become visible to reviewers, or that all non-arXived papers remain anonymous. Some papers may be arXived but under a different title, or may have been shared via other public channels (Twitter, personal websites, seminars) that also break anonymity. This measurement error in the treatment variable would tend to attenuate the estimated ATET toward zero, regardless of the NCO correction.

6. Limitations and Trade-offs

The Uncheckable Foundation: The NCO Assumption Cannot Be Verified

The entire NCO-based debiasing procedure rests on the assumption that long-term citation counts are not causally affected by early arXiving — formally, $N_a = N$ for $a = 0, 1$ (Assumption 3.1, point 4). The paper is unusually forthright about the status of this assumption, calling it "uncheckable" in the Limitations section and acknowledging that "the conclusions are contingent on this assumption." The causal graph (Figure 1) draws no arrow from $A$ (early arXiving) to $N$ (citation count), but this is an assertion, not an empirical finding — the data cannot distinguish between a graph where this arrow is absent and one where it exists but is small.

The consequences of this limitation are severe because the NCO assumption governs the direction of potential bias. If early arXiving does causally increase citation counts — through increased visibility, earlier indexing, flag-planting, or simply attracting attention from a wider audience — then the NCO contains part of the treatment effect signal. The DiD estimator $\mathbb{E}[Y_1 - N_1] - \mathbb{E}[Y_0 - N_0]$ would then subtract away not only quality confounding but also some genuine causal effect of arXiving on acceptance, biasing the ATET estimate downward — potentially masking a real effect that the paper then declares non-significant. The paper acknowledges the flag-planting concern (Goldberg, 2017) but dismisses it as lacking empirical evidence, noting that "we are skeptical that such an effect would be so widespread as to invalidate citation count as an NCO." This skepticism is reasonable but does not constitute evidence, and the possibility of an NCO violation cannot be ruled out from the available data.

The paper's primary defense against this limitation is the sensitivity analysis across citation-count windows ( $n \in \{1,2,3\}$ ) and binarization thresholds ( $q \in \{0.5, 0.75, 0.9\}$ ) shown in Figure 2. The authors argue that longer windows and higher thresholds produce "stronger" NCOs because they are less susceptible to causal effects of early arXiving. The pattern in Figure 2 is broadly consistent with this argument: ATET estimates shrink as $n$ increases (moving from top to bottom panels). For instance, at the 90% quantile, the estimated ATET falls from 7.56% ( $n=1$ ) to 5.13% ( $n=2$ ) to 2.16% ( $n=3$ ), with the $n=3$ estimate becoming non-significant. This monotonic relationship is suggestive — if the NCO were purely absorbing noise rather than confounding, we would not expect such a systematic reduction pattern — but it is not a formal validation. The $n=3$ sample retains only 570 matched pairs (versus 1,486 for $n=1$ ), so the loss of significance partly reflects reduced statistical power rather than only improved confounding control. A direct test of the NCO assumption — such as a randomized encouragement design where authors are randomly assigned to be nudged toward early arXiving, with downstream citation counts compared — does not exist and cannot be constructed from observational data.

The paper suggests no direct mitigation beyond the sensitivity analysis and the call for a randomized controlled trial. The QQ equi-confounding analysis (Appendix C.1) provides a robustness check under a different set of assumptions, producing an ATET of -4.375% with a confidence interval of (-9.965%, -0.092%) — consistent in significance level (weak-to-no evidence for a positive effect) but different in sign. This triangulation strengthens the conclusion that the effect is not large, but does not resolve the fundamental unverifiability of the NCO condition. The authors end the epilogue with an acknowledgment that captures the essence of the problem: "As with many real-world causal questions, assessing the true causal effect is hard." This is not a resolvable limitation of the study design but an inherent feature of observational causal inference with proxy variables — the most critical assumption is also the one least amenable to empirical verification.

Single-Venue, Single-Time-Period: Generalizability Is Unknown

All data come from a single conference — ICLR — over five years (2018–2022). The authors state this explicitly as a limitation: "we analyze decisions from a single machine learning conference where acceptance decisions are readily available, and thus our results may not generalize to other venues or research communities." The paper provides no empirical evidence about generalizability, because the dataset constraint that makes the study possible — ICLR publicly releases both accept and reject decisions for all submitted papers, while most other major ML conferences (NeurIPS, ICML, ACL, CVPR) do not — is precisely what prevents replication at other venues. The study is captive to the one conference where the necessary outcome data exist.

The consequences for external validity are substantial. ICLR differs from other major ML conferences along dimensions that could plausibly moderate the effect of early arXiving on acceptance:

Reviewer pool composition. ICLR reviewers skew toward deep learning and representation learning communities; conferences in other subfields (NLP at ACL, computer vision at CVPR, systems at OSDI) may have different norms around preprint posting and different susceptibilities to author-identity bias.
Acceptance rate. ICLR's acceptance rate fluctuated between roughly 25–35% during the study period. At more selective conferences (NeurIPS, ~20%) or less selective ones, the relationship between quality, author identity, and acceptance may differ — a fixed-quality paper that benefits marginally from arXiving at a moderately selective venue might see no benefit at a more selective one (where the quality threshold is higher and author identity may matter less at the margin) or a larger benefit at a less selective one.
ArXiving culture varies across subcommunities. NLP researchers, for example, have historically had a strong arXiving culture (the entire ACL anthology is openly accessible, and posting to arXiv is nearly universal). Computer vision researchers may have different norms. The ATET estimate from ICLR — which draws from multiple subcommunities — may not reflect the effect within any single subcommunity, where the "signal" of early arXiving (whether a reviewer infers author identity or quality from the mere fact of arXiving) could differ.
Temporal dynamics. The study period (2018–2022) saw rapid growth in both arXiv submissions and ML conference submissions. The "meaning" of early arXiving may have shifted over this period — in 2018, posting a preprint before the deadline might have been a strong signal of author confidence or paper readiness; by 2022, as arXiving became more routine, this signal may have weakened. The paper controls for year in the matching step, but this only ensures temporal balance, not temporal generalizability — the ATET estimated for 2018 may not equal the ATET for 2022, and the paper does not report year-stratified estimates that would reveal temporal trends.

The dataset limitation is not a flaw in the paper's design — the authors are using the only data available that permits a credible causal analysis — but it means the findings should be interpreted as evidence about ICLR specifically rather than about peer review generally. The authors acknowledge this explicitly ("our results may not generalize") but do not conduct sensitivity analyses that would help readers assess which features of ICLR are most likely to drive the findings and therefore which other venues might be expected to produce similar results. A practitioner considering whether to apply these findings to their own conference's anonymity policy would need to judge whether ICLR's review process is sufficiently similar to their own venue for the estimates to transfer — a judgment the paper does not assist with beyond the general caveat.

Quality Is Proxied, Not Measured — And the Proxy May Capture Constructs Beyond Quality

The paper's central methodological move is to use citation count as a proxy for the unobserved confounders collectively labeled "paper quality." The NCO framework then "differences out" the quality signal from the acceptance outcome. However, citation count captures more than paper quality as conventionally understood, and what it captures may overlap imperfectly with what affects acceptance decisions.

Citation counts are influenced by factors that are conceptually distinct from the quality that peer reviewers evaluate:

Author network effects: Well-connected authors may receive more citations through informal attention and citation cascades that are independent of paper quality per se.
Topic popularity: Papers on "hot" topics (e.g., large language models in 2022–2023) inherently attract more citations than equally meritorious papers on less fashionable topics, regardless of the paper's quality relative to its subfield.
Self-promotion and visibility: Authors who actively promote their work on social media, give talks, or release code may receive more citations for work of the same underlying quality as less-promoted papers.
Institutional prestige as a citation amplifier: Papers from highly-ranked institutions may be cited more frequently because readers assume quality based on institutional reputation — a mechanism that is itself a form of bias, and one that could create a correlation between the NCO and the very author characteristics the subgroup analyses are trying to assess.

Why does this matter for the NCO correction? The additive equi-confounding assumption (Assumption 4.1) requires that $U$ affects $Y$ (acceptance) and $N$ (citation count) identically. If the unobserved confounders affecting acceptance are primarily within-paper characteristics — clarity of exposition, technical novelty, correctness of proofs — while the confounders affecting citations include extra-paper characteristics — author networks, topic fashion, promotional effort — then the NCO is absorbing variance that is not purely quality, and the DiD correction may over-correct or under-correct depending on the relative magnitudes. Specifically, if citation count is more sensitive to author-network effects than acceptance decisions are (plausible, since citations accrue over years while acceptance is a one-time binary decision by a small set of reviewers), the NCO could remove author-network variance that is not confounding the acceptance outcome, potentially biasing the subgroup analyses (Figure 3) toward finding no differential effect across author groups.

The paper provides indirect evidence on this concern through the citation gap in Table B.2. For 2018 submissions in the matched sample, early-arXived papers have an average CC(3) of 249.95 versus 47.40 for matched controls — a 5.3× difference that persists after matching on author citation counts and institution ranks. This gap is so large that it is difficult to attribute entirely to non-quality factors like networks or promotion, especially since the matching already controls for the most obvious network-proxy variables (author citation counts, institution rank). The sheer magnitude of the citation gap suggests that a substantial quality signal is indeed present in the NCO. However, the paper does not decompose the citation gap into components or provide evidence that the quality-related portion of citation variance is the dominant driver of the NCO correction. The QQ equi-confounding analysis (Appendix C.1) partially addresses this by allowing $Y$ and $N$ to be affected by different confounders ( $U$ and $W$ respectively), requiring only that the distributional relationship with treatment is the same — which is a weaker assumption than requiring identical confounding structures. The qualitative consistency between the additive DiD and QQ results suggests that the NCO correction is not purely an artifact of conflating quality with network effects, but the uncertainty remains unquantified.

The authors do not claim citation count is a perfect quality measure — they explicitly note that "citation count is not to be confused with a measure of paper quality" (Section 4.2, footnote 9) — but the NCO framework's validity depends on the confounding structure being shared, not on the NCO being a direct measurement of the confounders. The distinction is subtle: the NCO does not need to measure quality; it only needs to share quality's confounding relationship with treatment and outcome. Citation counts influenced by non-quality factors can still serve this function if those non-quality factors are unrelated to early-arXiving behavior conditional on the observed covariates. The paper does not provide evidence on this conditional independence, and it represents a gap between the formal justification for the NCO and the real-world properties of the chosen proxy variable.

Sample Attrition Creates a Tradeoff Between NCO Credibility and Statistical Power

The paper's most credible NCO specifications — those using longer citation-count windows ( $n = 2$ or $3$ years) and higher binarization thresholds ( $q = 0.75$ or $0.90$ ) — are applied to progressively smaller analytic samples because recent conference years must be excluded when their n-year citation windows have not yet elapsed. Table B.1 shows the attrition: 1,486 matched pairs for $n=1$ (years 2018–2022), 1,073 pairs for $n=2$ (years 2018–2021), and 570 pairs for $n=3$ (years 2018–2020). This represents a 62% reduction in sample size between the largest and smallest analytic samples.

The consequence is a fundamental tension between assumption credibility and statistical power. The specifications where the NCO assumption is most defensible — three-year citation counts at a high threshold — are applied to fewer than 600 matched pairs, producing wide confidence intervals that make it difficult to detect a small but non-zero effect even if one exists. The pattern in Figure 2 illustrates this tension concretely:

At $n=1$ , $q=0.9$ : ATET = 7.56%, 95% CI excludes zero → significant , n = 1,486 pairs.
At $n=2$ , $q=0.9$ : ATET = 5.13%, 95% CI excludes zero → significant , n = 1,073 pairs.
At $n=3$ , $q=0.9$ : ATET = 2.16%, 95% CI includes zero → not significant , n = 570 pairs.

Is the loss of significance at $n=3$ due to better confounding control (the NCO is absorbing more quality signal) or reduced statistical power (the sample is too small to detect an effect of ~2%)? The confidence interval width increases noticeably from the $n=2$ to the $n=3$ panel in Figure 2, consistent with reduced power. The point estimate itself drops from 5.13% to 2.16%, which is consistent with better confounding control, but the two explanations are not distinguishable from the data. The paper's narrative treats the $n=3$ results as most credible ("Arguably, the strongest choice of NCO is an $N^{(n)}_q$ with a larger $n$ and higher $q$ ," Section 5.4), but the possibility that these "strongest" NCOs simply lack the power to detect an effect that is genuinely present cannot be ruled out.

The subgroup analyses (Figure 3) compound this problem. When the already-reduced $n=3$ sample is further stratified by institution rank or citation count, some cells contain as few as 2–6 treated papers (Tables B.3, B.4), yielding extremely wide confidence intervals that effectively preclude detecting any subgroup differences. The paper acknowledges this ("some strata have few samples, making it hard for statistical procedures to be sensible," Appendix B), but the acknowledgment does not solve the problem. The claim of "no differential effect across subgroups" — which is arguably the paper's most policy-relevant finding — is thus based in part on analyses where the statistical power to detect differential effects is minimal.

The paper offers no mitigation for this tradeoff beyond transparency about sample sizes. The tension is structural: the same temporal constraint that makes longer citation-count windows better NCOs (they are less affected by early arXiving) also reduces the available data. Future work might address this by extending the dataset to additional conference years as they become available, which would increase the $n=3$ sample without compromising the NCO's properties, but five years of data is all that currently exist.

This limitation interacts with the generalizability concern in a specific way: the $n=3$ sample draws exclusively from ICLR 2018–2020 — the earliest years in the dataset, and a period during which both arXiving norms and the ML research landscape differed substantially from 2021–2022 (the years excluded from the $n=3$ analysis). The COVID-19 pandemic, the explosive growth of large language model research, and shifting conference formats (virtual vs. in-person) all occurred during or after 2020. If the relationship between early arXiving and acceptance changed over this period — for instance, if early arXiving became more common and thus less informative as a quality signal — the $n=3$ estimates would reflect only the pre-2021 regime and might not characterize the current state of peer review.

Matching Balances Observables but May Not Balance What Matters for Selection Into Treatment

The tripartite matching procedure (Section 4.1, Table A.1) achieves excellent balance on all 18 observed covariates, with post-matching standardized mean differences below 0.04 for all variables. This is a strength of the study design — it eliminates overt compositional differences between treated and control groups on the variables the authors could measure. However, the very success of the matching on observables raises a question the paper does not fully address: are these the right covariates for modeling selection into early arXiving?

The decision to arXiv a paper before the review deadline is not random — authors actively choose whether to post, and that choice likely reflects factors beyond the 18 covariates in Table 1. Consider what drives an author to arXiv early: confidence in the paper's readiness, desire for feedback before the deadline, norms within their subcommunity, career stage (junior researchers may be more cautious about publicly posting work that could be rejected), prior experience with peer review, institutional culture around preprinting, and strategic considerations about establishing priority. Some of these factors — particularly confidence and perceived paper readiness — are plausibly correlated with the paper's actual quality and thus with acceptance. The matching procedure does not (and cannot) balance on these unobserved determinants of treatment selection, because they are not in Table 1.

The consequence is that even after matching, the treated and control groups may differ on the propensity to arXiv early, not just on the covariates. A paper whose authors arXived early is, almost by definition, a paper whose authors believed it was ready for public scrutiny before the conference deadline. A matched control paper — observationally similar on all 18 covariates but not arXived early — may be a paper whose authors were less confident, had different norms, faced different institutional pressures, or simply forgot to post. The ATET compares these two groups and attributes the difference in acceptance rates (after NCO correction) to the causal effect of arXiving. But if the unobserved determinants of arXiving behavior are themselves correlated with acceptance — say, author confidence predicts both arXiving and careful polishing, and careful polishing increases acceptance chances — then the ATET estimate still contains residual selection bias that neither matching nor the NCO addresses.

This is not a failure of the matching procedure per se (no matching algorithm can balance on unmeasured variables), but it is a limitation of the study design that the paper underemphasizes. The causal graph (Figure 1) draws an arrow from $U$ (unobserved confounders) to $A$ (treatment), acknowledging that unobserved factors influence arXiving decisions. The NCO framework is designed to correct for confounding of the $A \rightarrow Y$ relationship, but it does not address selection bias in the sense of systematic differences between units that choose treatment versus those that do not — the ATET itself is defined over the treated subpopulation, and the counterfactual $Y_{A=0} \mid A=1$ (what would have happened to arXived papers had they not been arXived) must be estimated from control papers that chose not to arXiv. If the decision to arXiv reflects unmeasured paper or author characteristics that also affect acceptance independent of the arXiving action itself, the control group is an imperfect counterfactual even after matching and NCO correction.

The paper's defense against this concern is implicit in the matching design: by balancing on author citation counts, institution ranks, and text features, the matching attempts to make the control group as similar as possible to the treated group on dimensions that might correlate with both the arXiving decision and acceptance. The excellent post-matching balance (Table A.1) shows this succeeds for the measured dimensions. But the 5.3× citation gap in Table B.2 — where matched controls are observationally identical on 18 covariates yet dramatically less cited — demonstrates that substantial unmeasured differences remain. The NCO absorbs some of this residual variation, but whether it absorbs the specific variation related to why authors arXiv early (as distinct from intrinsic paper quality) is unknown and untested.

No experiment in the paper directly addresses this limitation. A randomized controlled trial — which the authors advocate in the conclusion — would solve it by design, since randomization breaks the link between the decision to arXiv and all baseline characteristics (observed and unobserved). In the absence of randomization, the limitation is inherent to the observational study design and cannot be eliminated, only acknowledged.

The NCO Binarization Thresholds Are Arbitrary and Influence the Results

The paper constructs its NCO variables by thresholding raw citation counts at the $q$ -th empirical quantile of the matched sample, for $q \in \{0.5, 0.75, 0.9\}$ . These thresholds are described as motivated by the need to put $Y$ (binary acceptance) and $N$ (citation count) on the same scale to satisfy the additive equi-confounding assumption (Section 4.2, Appendix D). However, the specific choice of $q$ is arbitrary — there is no principled argument for why 50%, 75%, or 90% are the "right" thresholds, and the results in Figure 2 show that the ATET estimate varies systematically with $q$ .

At $n=3$ , the ATET ranges from -9.17% ( $q=0.5$ ) to -2.63% ( $q=0.75$ ) to 2.16% ( $q=0.9$ ). The point estimate changes sign and more than doubles in magnitude across the three thresholds. Only the $q=0.9$ specification is discussed as "strong" in the main text, but the $q=0.5$ specification — which uses the median as the threshold — produces a large negative point estimate (-9.17%) that the paper does not devote sustained attention to, beyond noting it is not statistically significant. The $q=0.75$ specification produces an intermediate value (-2.63%).

The consequence is that the paper's headline conclusion — "early arXiving may have a small effect" — depends in part on which thresholds the reader accepts as valid. A reader who believes the median threshold ( $q=0.5$ ) is the correct operationalization would conclude the effect might be negative and substantial in magnitude (-9.17%), not small and positive. A reader who believes the $q=0.9$ threshold is correct would conclude the effect is small and positive (2.16%) but not statistically distinguishable from zero. The paper provides a qualitative argument for preferring higher $q$ ("stronger" NCOs) but does not provide a formal criterion for threshold selection or a procedure for combining estimates across thresholds.

Why is this not resolved by the sensitivity analysis across $q$ ? The sweep demonstrates that the choice matters — the ATET is not invariant to the threshold — but it does not provide guidance on which threshold is most appropriate. The authors' argument that higher $q$ produces a "stronger" NCO is based on the claim that extremely highly-cited papers (top 10%) are less likely to have their citation status causally affected by early arXiving than moderately cited papers (top 50%). This is plausible — being in the top 10% of cited papers likely reflects sustained impact that early arXiving alone cannot manufacture, whereas being above the median might be achievable through a modest visibility boost — but it remains an intuitive argument rather than an empirical demonstration. A paper that would have been at the 52nd percentile of citations without arXiving might be pushed above the 50th percentile threshold by a small arXiving-induced visibility boost, making the $q=0.5$ NCO more susceptible to treatment-effect contamination. But the paper cannot test this hypothesis because it cannot observe the counterfactual citation counts for arXived papers.

The QQ equi-confounding analysis (Appendix C.1) addresses this limitation directly by using raw (un-binarized) citation counts, eliminating the threshold choice entirely. The QQ result (ATET = -4.375%, CI: [-9.965%, -0.092%]) is broadly consistent with the $n=3$ , $q=0.5$ and $q=0.75$ DiD estimates in suggesting a near-zero or negative effect, and less consistent with the $q=0.9$ DiD estimate of +2.16%. This sensitivity to the operationalization of the NCO — where different reasonable choices produce ATET estimates ranging from -9.17% to +2.16% on the same $n=3$ sample — means a practitioner making decisions based on these results needs to either select a threshold based on assumptions about the NCO's properties (and accept the fragility that entails) or combine the estimates across thresholds in some principled way. The paper does the former implicitly (favoring high- $q$ specifications) and does not attempt the latter.

The authors acknowledge that threshold choice is a limitation: "we do not know what is the 'correct' threshold for dichotomizing" and "much information was lost during this process" (Appendix D). But they frame these as motivation for the QQ sensitivity analysis rather than as a threat to the validity of the main DiD results. A reader who finds the $q=0.9$ threshold most defensible — as the authors do — will reach the paper's headline conclusion. A reader who finds the median threshold or the raw-citation QQ approach more defensible will reach a somewhat different conclusion (the effect may be negative or zero, not small-positive). This ambiguity is not fully resolved by the paper's analysis, and it represents a degree of freedom in the estimation procedure that could, in principle, be exploited to support different narrative framings of the same underlying data.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new algorithm, model, or causal inference method that would disrupt the methodological state of the art. What it does — and this is a contribution of a different kind, no less significant for its nature — is redirect the burden of proof in a consequential, years-long policy debate that had previously been litigated on the basis of intuition and anecdote. The effect is to transform the question "does early arXiving create unfair advantage?" from a matter of first-principles argumentation into an empirically tractable causal inference problem with an explicit, assumption-transparent analysis that has now been conducted and peer-reviewed.

The magnitude of the shift is worth calibrating. This is not a paradigm shift in the Kuhnian sense — the paper does not overturn an established theoretical framework or introduce a new research program. It is better understood as a diagnostic reframing with policy consequences. Before this work, the default position in the ACL policy community (and arguably in the broader ML conference ecosystem) was that early arXiving probably advantages well-known authors, that the risk of such advantage justified precautionary restrictions (the anonymity period), and that the absence of direct causal evidence did not weaken this logic because the stakes were high and the restriction was relatively low-cost. The paper systematically challenges each element of this implicit reasoning: it finds no evidence for differential advantage across author subgroups (Figure 3), estimates the overall treatment effect as small and often non-significant after quality deconfounding (Figure 2), and demonstrates that the apparent ~10% acceptance advantage in naive analyses is largely attributable to quality differences that the anonymity-period debate had not adequately accounted for.

This reframing operates at two levels. At the level of specific evidence, the paper provides the first credible causal estimate of the early-arXiving effect on acceptance, giving policy-makers a concrete number (under 4% in most specifications, insignificant in four of nine) rather than a directional worry. The fact that the ACL removed its anonymity period on the same day this paper was accepted — and roughly six months after the preprint appeared — does not establish causation, but it illustrates the practical consequence of having empirical evidence where previously there was only speculation. Whether or not this paper influenced the ACL's decision, the next time a conference considers imposing or removing an anonymity period, the default reference point will no longer be "prior work is mixed and inconclusive" but rather "the only direct causal analysis of early-arXiving effects found them to be small and non-differential."

At the level of methodology, the paper provides a template for applying the NCO framework to questions of evaluation fairness — a class of problems that extends well beyond peer review to grant funding, hiring committees, award selections, and any setting where evaluators' decisions may be influenced by both legitimate quality signals and illegitimate identity-based biases, with quality being difficult to measure directly. The specific move — using a post-decision outcome variable (citations, in this case) that shares quality confounders with the evaluation outcome but is not causally affected by the treatment — is portable to any domain where such a proxy exists. The paper also demonstrates that conducting the analysis under multiple incomparable identification assumptions (additive DiD vs. QQ equi-confounding) and checking for convergence in conclusions is a practical way to build credibility when no single set of assumptions is clearly correct. For the growing community of NLP researchers applying causal methods to observational text data (Feder et al., 2022), this paper serves as a worked example of how to structure such an analysis from causal graph to estimator to robustness checks.

The paper also resolves a latent contradiction in prior work that had not been explicitly articulated as a contradiction. The prior literature contained both claims that double-blind review reduces bias (Tomkins et al., 2017, finding single-blind review at WSDM favored highly-ranked institutions) and claims that double-blind review makes no difference (Madden and DeWitt, 2006, finding no bias at SIGMOD). These findings were not exactly contradictory — they studied different venues with different review processes and different author populations — but they created an impression of inconsistency that made it easy for both sides of the anonymity-period debate to cite supportive evidence. The present paper does not directly test double-blind versus single-blind review (it studies early arXiving as a mechanism that de facto breaks double-blindness for some papers), but it provides a framework for understanding why prior studies diverged: the effect of de-anonymization likely depends on venue-specific factors (reviewer pool, acceptance rate, arXiving norms) and on the specific mechanism by which anonymity is broken (early arXiving vs. formal single-blind review). The paper's null finding on differential subgroup effects suggests that for ICLR, at least, the feared mechanism — reviewers using author identity to favor high-status authors — either does not operate or operates too weakly to detect. This does not contradict Tomkins et al.'s finding of bias at WSDM, but it narrows the conditions under which such bias might be expected: perhaps WSDM's single-blind process produced different effects than ICLR's de-facto de-anonymization through arXiving, or perhaps the bias at WSDM was driven by factors not captured by the institution-rank and citation-count stratifications used here. Future work that replicates the present study's design at other venues would directly test these possibilities.

Finally, the paper's finding that quality confounding is large — the 5.3× citation gap in Table B.2 between arXived and matched non-arXived papers — has implications beyond the specific treatment effect estimate. It suggests that any associational study of peer review outcomes that does not account for quality confounding is likely to be severely biased, because papers that authors choose to make public early differ from those they do not on dimensions that dramatically affect downstream impact. This is a cautionary finding for the broader science-of-science literature, where associational analyses of author characteristics and publication outcomes remain common. The paper's diagnostic move — showing the citation gap as evidence that residual confounding is large even after exhaustive covariate matching — provides a template that other observational studies of peer review could adopt to assess the plausibility of their own unconfoundedness assumptions.

Follow-Up Research This Work Enables

Replicating the NCO design at other conferences with full outcome data. The most direct extension is to apply the identical causal framework to any other conference that publicly releases both accept and reject decisions for all submitted papers. At present, ICLR appears to be unique among major ML venues in doing so, but several smaller workshops and some non-ML fields (e.g., certain biomedical informatics venues) have similar data-release policies. A replication at a venue with different reviewer demographics, acceptance rates, and arXiving norms would directly test the generalizability of the paper's null findings. The key measurement needed is the ATET under the same nine NCO specifications, plus the subgroup analyses stratified by institution rank and citation count. If a second venue produced a large, significant, positive ATET — especially one that differed across author subgroups — it would demonstrate that the ICLR results are venue-specific and that anonymity periods may be justified at some venues but not others. Conversely, if multiple venues produced consistently small and non-significant effects, the case for anonymity periods as a fairness intervention would be substantially weakened. The limiting factor is data availability, not analytical complexity: the entire pipeline (matching, NCO construction, DiD estimation) is implemented and released, so replication at a new venue would be primarily an exercise in data collection and S2 API querying.

Training a difficulty estimator for arXiving propensity — or its equivalent. The paper's matching procedure balances on observed covariates but cannot balance on the unobserved determinants of the decision to arXiv early, such as author confidence, subcommunity norms, or strategic considerations about priority. A natural follow-up would be to collect survey or interview data from a sample of ICLR authors about their arXiving decisions: why did they choose to arXiv early (or not), what factors influenced their timing, and how did they perceive the risks and benefits? These qualitative data could be used to construct additional covariates that capture the selection-into-treatment process more directly. For example, an author's self-reported confidence in the paper's acceptance chances, or their lab's explicit policy about preprinting, could be included as matching covariates in a future iteration of the study, reducing residual selection bias. Even without a full survey, one could construct a proxy for arXiving propensity by training a classifier to predict treatment status from paper and author features, and then using the predicted propensity as an additional matching covariate or as the basis for propensity-score-based sensitivity analyses (e.g., Rosenbaum bounds) that quantify how much unobserved confounding would be needed to explain away the residual ATET.

A randomized encouragement trial at a major conference. The paper's most emphatic recommendation — stated in both the Discussion and the Limitations — is a randomized controlled trial. The design would involve randomly assigning submitting authors to one of two conditions: (1) encouraged to arXiv their paper before the review deadline, or (2) asked to refrain from arXiving until after the decision. Because forcing authors to arXiv or not arXiv is ethically and practically infeasible, the trial would be an encouragement design, where the randomization affects the encouragement but authors remain free to comply or not. The primary analysis would estimate the complier average causal effect (CACE) using the random assignment as an instrumental variable. The outcome would be the acceptance decision, and the analysis would be stratified by author subgroups to test for differential effects. The key measurements: (a) the first-stage effect of encouragement on actual arXiving behavior (to assess compliance), (b) the ITT (intention-to-treat) effect of encouragement on acceptance, and (c) the CACE for compliers. The trial would require cooperation from a conference with sufficient submissions to power the subgroup analyses — ICLR or NeurIPS would be natural candidates — and would need to navigate the ethical dimension of randomly encouraging some authors to forgo early publicity for their work. The paper's empirical estimates provide a basis for power calculations: if the true ATET is under 4%, detecting it with 80% power at α = 0.05 would require a very large sample (thousands of papers per arm), suggesting that a trial might be powered to rule out effects above some threshold (e.g., 5 percentage points) rather than to precisely estimate a small effect. A trial that found an effect below this threshold — say, an ATET of less than 3 percentage points with a tight confidence interval — would effectively settle the policy question by establishing that any effect is too small to justify restricting authors' dissemination freedom, even if it is non-zero.

Extending the NCO framework to other evaluation settings with available post-decision outcomes. The paper's methodological template — use a post-decision outcome that shares confounders with the evaluation decision but is not causally affected by the treatment — applies to any setting where evaluators make binary decisions (accept/reject, fund/do-not-fund, hire/do-not-hire) and where some downstream measure of the evaluated entity's "true quality" is available. Concrete candidates include: (a) Grant funding: does knowing the applicant's identity (via pre-proposal publicity, personal connections, or non-anonymized review) affect funding decisions, using subsequent publication and citation records of funded vs. unfunded proposals as the NCO? This would require data from a funding agency that releases both funded and unfunded proposal outcomes (rare, but some agencies do so in anonymized form). (b) Faculty hiring: does the visibility of a candidate's work (via arXiv preprints, social media presence, conference talks) affect hiring decisions at research universities, using the candidate's subsequent publication record as the NCO? This would require data on short-listed candidates and hiring outcomes, which is sensitive but potentially obtainable through institutional partnerships. (c) Conference award selection: do best-paper awards favor work from well-known authors or prestigious institutions, using the awarded papers' subsequent citation counts as the NCO? This is a lower-stakes version of the peer-review question and could be studied with publicly available award data from multiple conferences. Each of these extensions would require careful justification of the NCO assumption in the new context — does the post-decision outcome really share confounders with the evaluation decision, and is it truly unaffected by the treatment? — but the basic analytical machinery (matching + DiD + NCO) transfers directly.

Decomposing the citation gap to quantify how much of the NCO correction is driven by quality versus other factors. The 5.3× citation gap between arXived and matched non-arXived papers (Table B.2) is presented as evidence of quality confounding, but the paper does not decompose this gap into components. A follow-up study could attempt to predict citation counts from paper features that are plausibly orthogonal to quality — author network size (co-authorship graph centrality), topic "hotness" (growth rate of the paper's topic area), promotional activity (Twitter mentions, conference presentations), institutional prestige independent of the specific paper — and residualize the citation count on these predictors. The residual citation count (net of network, topic, and promotion effects) would be a "purer" NCO that captures quality more directly, and using this residualized NCO in the DiD estimator would produce an ATET estimate that is less susceptible to the concern that the NCO is absorbing non-quality variance. If the ATET estimate remains small and non-significant with the residualized NCO, it would strengthen the paper's conclusion by addressing the limitation discussed in Section 6 about the NCO capturing constructs beyond quality. If the ATET becomes larger and significant, it would suggest that the original NCO was over-correcting by absorbing non-quality variance that is not shared with acceptance decisions, and that a genuine arXiving effect had been masked.

A dynamic or adaptive arXiving policy analysis using time-varying treatment definitions. The paper defines treatment as a binary variable: arXiv before the reviewing deadline or not. In practice, the timing of arXiving relative to the deadline varies — some papers are posted months before, others days before. A more granular treatment definition could distinguish between "very early" arXiving (e.g., >3 months before the deadline), "moderately early" arXiving (1–3 months), and "just-in-time" arXiving (<1 month), under the hypothesis that very early posting provides more opportunity for reviewers to discover and be influenced by author identity. This would convert the binary treatment into a multi-level or continuous treatment, requiring a generalization of the DiD + NCO estimator to multiple treatment levels. The analysis would answer a subtler policy question: if early arXiving does affect acceptance, is the effect concentrated among papers posted far in advance (suggesting a visibility mechanism where reviewers have time to look up the authors) or among papers posted just before the deadline (suggesting a signaling mechanism where the mere fact of arXiving conveys information)? The arXiv posting dates are available from the S2 data and would not require additional data collection, only a more sophisticated causal model. A null finding — no effect at any posting timing — would further support the paper's conclusion; a finding that very-early posting has a larger effect than near-deadline posting would suggest that the treatment effect varies with the "dose" of visibility, providing more nuanced guidance for policy (e.g., an anonymity period that only restricts arXiving in the months immediately preceding the deadline, rather than a blanket restriction).

Practical Applications and Downstream Use Cases

Conference policy-making for anonymity periods. This is the most direct and consequential application. Program chairs, steering committees, and professional societies (ACL, NeurIPS, ICML, AAAI) that are considering whether to institute, modify, or remove an anonymity period now have a peer-reviewed, publicly available causal analysis to inform their decision. The paper's key empirical finding — that early arXiving does not confer a statistically distinguishable advantage to any author subgroup, and that the overall effect on acceptance is small (under 4% in most specifications) — provides evidence-based grounds for concluding that anonymity periods may not be necessary to preserve fairness across researcher populations. In the paper's own words (Section 6): "our findings call this justification into question." A conference that removes its anonymity period can cite this paper as evidence that the feared unfairness does not materialize, at least under the assumptions and within the venue (ICLR) studied. Conversely, a conference that retains its anonymity period can point to the paper's acknowledged limitations — single-venue, uncheckable NCO assumption, limited statistical power for subgroup analyses — as reasons for continued caution. The paper's clear articulation of its own limitations makes it usable by both sides of the debate, but its existence shifts the default from "we don't know, so let's be cautious" to "the best available evidence suggests small and non-differential effects, so the burden is on those claiming a large or unfair effect to produce evidence."

Author decision-making about whether and when to arXiv. Individual researchers deciding whether to post their paper to arXiv before a submission deadline can use the paper's estimates to inform their decision calculus. The finding that early arXiving increases acceptance chances by at most a few percentage points (and possibly not at all, given the insignificance in many specifications) suggests that the strategic benefit of arXiving early for acceptance purposes is minimal. Authors who were delaying arXiving out of fear that it would hurt their chances (by revealing their identity to potentially biased reviewers) can take some reassurance from the subgroup analyses showing no differential disadvantage for low-citation or low-ranked-institution authors. Authors who were arXiving early specifically hoping for an acceptance boost should not expect a large effect, and the decision should instead be driven by the other benefits of early arXiving — getting feedback, establishing priority, making the work visible to the community — which are substantial and unaffected by the paper's findings. A caveat: the paper's estimates are for ICLR specifically; authors submitting to other venues should consider whether their target conference's review process is similar enough for the findings to transfer.

Designing self-improvement pipelines for conference submission systems. This application reframes the paper's analysis as a diagnostic tool. Conference management systems (OpenReview, CMT, EasyChair) could, in principle, run the paper's matching and NCO analysis internally on their own submission data (which includes both accept/reject decisions and paper metadata) to generate venue-specific estimates of the early-arXiving effect. These estimates could be reported to program chairs as part of the post-conference analytics package, alongside acceptance rates by topic and author demographics. The analysis would not require the S2 citation-count step for a basic associational estimate (the Unadj analysis), and if the conference has access to citation data (e.g., through a partnership with Semantic Scholar or Google Scholar), the full NCO analysis could be replicated internally. This would address the generalizability limitation directly: each venue would know its own early-arXiving effect estimate, rather than relying on the ICLR-specific numbers reported here. If a conference ran this analysis over multiple years and consistently found a near-zero effect, it would build a longitudinal evidence base that is more powerful than any single-year analysis. The computational cost is minimal (matching on a few thousand submissions takes seconds; S2 API queries are the bottleneck), and the output would be directly actionable for policy decisions.

Informing the design of double-blind review in non-conference settings. The paper's finding that early arXiving does not create differential advantage across author subgroups has implications beyond conference peer review. Journal review processes (which typically have longer timelines and different arXiving norms), grant review panels (where applicants' identities are often known with or without preprints), and internal promotion and hiring committees in research organizations all face analogous questions about whether pre-decision publicity advantages certain candidates. While the paper's specific estimates do not transfer directly to these settings, the analytical framework — matching on observed confounders, using a post-decision outcome as an NCO to absorb quality-related confounding, stratifying by author subgroups — provides a replicable template. A research organization concerned about bias in its internal review processes could adapt the paper's pipeline to its own data, substituting an appropriate NCO for the setting (e.g., subsequent project success metrics for grant review, post-hire productivity for faculty hiring). The paper's demonstration that the NCO correction substantially reduces the estimated treatment effect — from ~10% to under 4% — serves as a caution that naive associational analyses in these other settings likely overstate the magnitude of any bias, and that quality confounding must be addressed before claiming a causal effect of identity revelation on evaluation outcomes.