How AI Impacts Skill Formation

ArXiv: 2601.20245

🎯 Pitch

Developers who used an AI coding assistant to learn a new programming library learned 17% less than those who struggled on their own — and weren’t any faster at completing tasks. The critical difference wasn’t whether they used AI, but how: those who stayed cognitively engaged by asking conceptual questions or understanding generated code kept their learning gains, while those who fully delegated lost out entirely.

1. Executive Summary

This paper studies how AI assistance affects skill formation when knowledge workers acquire new technical competencies, conducting randomized controlled experiments in which professional developers learn a new asynchronous Python library (Trio) either with or without access to a GPT-4o–powered coding assistant. The central finding is that AI use impairs conceptual understanding, code reading, and debugging abilities — producing a statistically significant 17% reduction in evaluation scores (Cohen's d=0.738, p=0.010) — without delivering significant efficiency gains on average, largely because participants who remained cognitively engaged (for example, by asking only conceptual questions or requesting explanations alongside generated code) preserved learning outcomes despite using AI, while those who fully delegated coding tasks experienced the largest skill erosion. Through qualitative analysis of screen recordings, the paper identifies six distinct AI interaction patterns and demonstrates that three of them — Generation-Then-Comprehension, Hybrid Code-Explanation, and Conceptual Inquiry — yield high quiz scores (65–86%) by maintaining cognitive effort, establishing that AI-enhanced productivity is not a shortcut to competence and that skill preservation depends critically on how the tool is used rather than whether it is used.

2. Context and Motivation

The Core Problem: AI Productivity Gains May Come at the Cost of Human Expertise

The fundamental question this paper tackles is deceptively simple: when workers use AI assistance to complete tasks that require new skills, do they actually learn those skills? This matters because the prevailing narrative around AI in the workplace emphasizes dramatic productivity improvements — particularly for novice workers — without accounting for what happens to the workers themselves in the process. The paper identifies a tension that has been largely ignored: the very populations that benefit most from AI assistance (junior developers, less experienced workers) are also the ones who most urgently need to develop foundational expertise.

This gap is significant for several interconnected reasons the authors develop throughout Sections 1–3:

Supervision requires competence. As AI systems become more capable and are deployed in higher-stakes settings, humans remain legally and ethically responsible for the quality of AI-generated outputs. Debugging, validating, and supervising AI-written code requires genuine technical understanding — the kind that may be compromised if workers use AI as a crutch during their formative learning experiences. As the authors frame it in Section 1:

"In high-stakes applications, AI written code may be debugged and tested by humans before a piece of software is ready for deployment. This additional verification that enhances safety is only possible when human engineers themselves have the skills to understand code and identify errors."

The automation paradox in skill acquisition. The historical pattern of automation — from factory robots to accounting software — shows that humans shift from performing tasks to supervising them. But this supervision role presupposes that humans retain the technical knowledge to identify errors when automation fails. If AI assistance during the learning phase erodes this foundational knowledge, we risk creating a generation of workers who can prompt AI effectively but cannot evaluate its outputs critically. This connects directly to the "scalable oversight" problem the authors cite: as AI systems become more sophisticated, supervising them requires humans who understand what correct outputs look like — yet AI tools may be actively undermining the development of exactly these evaluation skills.

Why Prior Work Leaves This Question Unanswered

The paper identifies a specific gap in the empirical literature: while many studies document AI's productivity effects, almost none examine how AI affects the process of skill acquisition during task completion. The authors make this distinction explicit in their introduction:

"Although most studies have focused on the end product of AI assistance (e.g., lines of code written, quality of ideas proposed), an equally important, if not more crucial question is how process of receiving AI assistance impacts workers."

This product-versus-process distinction is the paper's central analytical lens, and it reveals several limitations in prior work.

Productivity Studies: Measuring Output, Ignoring Development

The paper acknowledges a robust body of evidence showing that AI improves task completion efficiency. Call center workers resolve more issues per hour (Brynjolfsson et al., 2025), consultants complete more tasks (Dell'Acqua et al., 2023), and software developers write code faster — one study found a 55.5% speed improvement with Copilot (Peng et al., 2023), while another documented a 26.8% boost in productivity metrics like pull requests and commits (Cui et al., 2024). Critically, these productivity gains are consistently largest for novice or less experienced workers — the exact population most in need of skill development.

The paper's concern is not that these productivity findings are wrong, but that they are incomplete. They measure what workers produce during a study but not what workers retain afterward. A junior developer who completes tasks twice as fast with AI may also learn half as much — a tradeoff invisible to productivity metrics but catastrophic for long-term professional development. The paper positions itself as filling this gap by measuring both productivity (task completion time) and skill formation (quiz performance) in the same experiment, enabling a direct test of whether AI creates a tradeoff between immediate efficiency and lasting competence.

Cognitive Offloading and Skill Retention: Suggestive but Not Causal

The paper identifies two adjacent lines of research that point toward its central concern but stop short of providing causal evidence about skill acquisition in professional settings:

Cognitive offloading research has documented that AI tool use is associated with reduced critical thinking and increased dependence. For instance, Gerlich (2025) found that frequent AI use among knowledge workers correlated with worse critical thinking abilities, and Lee et al. (2025) found that knowledge workers reported lower cognitive effort and confidence when using generative AI. The paper notes, however, that these are observational surveys — they cannot establish causality. Workers who already have weaker critical thinking skills might self-select into heavier AI use. The paper's randomized controlled design directly addresses this limitation by randomly assigning participants to AI or no-AI conditions, enabling causal claims about AI's effect on skill formation that survey research cannot support.

Skill retention studies have examined what happens after AI assistance is removed. Wu et al. (2025) found that even when generative AI improved immediate performance on content creation tasks, the improvement did not persist when humans later performed similar tasks independently. Wiles et al. (2024) described this effect for data science tasks as an "exoskeleton" — the enhanced capabilities AI provided to non-technical consultants disappeared once AI access was withdrawn. The paper's contribution extends this line of work by asking: does AI not only fail to transfer skills, but actively impair their acquisition in the first place? The authors frame their work explicitly as a natural follow-up to Wiles et al.:

"Our work asks the natural follow-up question of whether the usage of AI tools could cause worse learning outcomes for the acquisition of skills on the job for technical professionals themselves."

This is a crucial distinction. Skill retention research shows that AI-generated capabilities don't survive the removal of AI. This paper asks whether AI use actually prevents skills from forming during the learning process — a more fundamental concern for workforce development.

CS Education Research: Observing but Not Experimenting

The paper draws on computer science education literature that has observed how students use AI tools in learning contexts, but notes that these studies are almost entirely observational. Poitras et al. (2024) tracked students over a semester and found that those with less coding proficiency were more likely to seek AI assistance — a correlation that could reflect either causation (AI hinders learning) or selection (struggling students seek more help). Prasad et al. (2023) coded different ways students used LLMs in coursework and found that upper-year students relied less on AI assistance, but this doesn't tell us whether AI caused differences in learning or simply reflected different levels of existing competence.

The paper's contribution is to bring the rigor of randomized experimentation to these questions. By randomly assigning participants to AI or control conditions, the authors can isolate the causal effect of AI assistance on learning outcomes — something the observational education literature cannot do.

Overreliance Literature: Focused on Decisions, Not Development

A substantial body of work on "overreliance" examines how humans make decisions after receiving erroneous AI recommendations (Buçinca et al., 2021; Vasconcelos et al., 2023; Klingbeil et al., 2024). This literature has developed interventions to reduce overreliance, such as providing explanations (Vasconcelos et al., 2023) or dissenting opinions (Reingold et al., 2024). The paper acknowledges this work but identifies a fundamental difference in scope: the overreliance literature focuses on decision-time behavior — how humans evaluate AI outputs at the moment of interaction — while the paper focuses on developmental trajectories — how AI interaction patterns during learning tasks shape what humans know and can do afterward. The overreliance framework asks "do humans trust AI when they shouldn't?" The paper asks "does using AI prevent humans from developing the expertise to know when they shouldn't trust it?"

How This Paper Positions Itself

The paper fills a specific, previously unaddressed intersection: the causal effect of AI assistance on skill formation during on-the-job learning. It bridges the productivity literature (which measures output), the cognitive offloading literature (which is observational), the skill retention literature (which examines what happens after AI is removed), and the overreliance literature (which focuses on decision-time behavior) by providing the first randomized controlled experiment that measures both productivity and skill development simultaneously for a professional technical task.

The framing draws on experiential learning theory — specifically Kolb's experiential learning cycle and Problem-Based Learning (Schmidt, 1994) — to conceptualize AI assistance as a "shortcut" that bypasses the learning stage of task completion. In the paper's Figure 2, learning without AI follows a path through task engagement, encountering and resolving errors, and building understanding through struggle. AI assistance offers an alternative path that reaches task completion without these developmental experiences. The research questions operationalize this framing: RQ1 tests whether the shortcut improves productivity, while RQ2 tests whether it comes at the cost of skill formation.

The paper is careful not to position itself as anti-AI. Instead, it identifies modes of AI interaction that maintain cognitive engagement while still providing assistance, suggesting that the problem is not AI use per se, but specific patterns of passive reliance. This is evident in the qualitative analysis (Section 6), where the authors distinguish between "low-scoring interaction patterns" (AI Delegation, Progressive AI Reliance, Iterative AI Debugging) and "high-scoring interaction patterns" (Generation-Then-Comprehension, Hybrid Code-Explanation, Conceptual Inquiry), demonstrating that skill preservation depends on how AI is used, not just whether it is used.

The choice of software engineering as the study domain is deliberate and well-justified. The authors identify it as a profession where AI tools are already rapidly adopted, where productivity gains for novice workers are well-documented, and — critically — where the combination of persistent human liability for errors and demonstrated AI productivity creates exactly the tension the paper studies. The Trio library is chosen because it is real but unfamiliar (ensuring the task genuinely requires new skill acquisition), explicitly designed to be easy to learn (making a one-hour experiment feasible), and involves conceptual novelty (structured concurrency) beyond mere Python syntax (ensuring the skills being measured are genuine learning targets rather than superficial pattern matching).

3. Technical Approach

3.1 Reader Orientation

The paper constructs an experimental apparatus to measure both how quickly programmers complete an unfamiliar coding task and how deeply they understand the new technical concepts afterward — simultaneously. It solves the problem of isolating AI's causal effect on skill formation by randomly assigning professional developers to either use a GPT-4o–powered coding assistant or not while learning the Trio asynchronous programming library, then measuring both task completion speed and a comprehensive quiz score on conceptual understanding, code reading, and debugging that was designed and psychometrically validated through four pilot studies.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental system has five major components:

Participant screening and balancing — a pre-recruitment survey that ensures all participants have Python experience, have used AI coding tools before, and have never used the Trio library, with balanced assignment across experience levels, prior asynchronous programming knowledge, and Python usage frequency.
Warm-up calibration task — a simple synchronous Python problem (adding a border around a list of strings, averaging 4 minutes) completed by all participants without AI access to calibrate baseline Python fluency and familiarize participants with the coding platform interface.
Main Trio learning tasks — two coding problems using the unfamiliar Trio asynchronous library with a 35-minute time limit, where participants are randomly assigned to either the treatment condition (access to a GPT-4o chat-based AI assistant embedded in the coding platform alongside web search and task instructions) or the control condition (web search and task instructions only, with an enforced no-AI pledge verified through screen recordings).
Comprehensive skill evaluation — a 14-question, 27-point quiz covering debugging, code reading, and conceptual understanding across 7 core Trio concepts, administered to all participants without AI access and designed through item response theory testing across five quiz versions in preliminary studies.
Screen recording qualitative analysis — manual frame-by-frame annotation of every participant's screen recording to code interaction patterns, error encounters, query types, AI adoption methods, and active coding time, yielding the six interaction personas and the error-frequency analysis.

Information flows as follows: screened and balanced participants arrive → all complete the warm-up task without AI → random assignment to treatment or control → both groups attempt the two Trio tasks within 35 minutes (treatment group has the AI assistant; control group has pledged not to use AI and this is verified by screen recording) → all participants take the quiz without AI access → all participants complete a post-experience survey → researchers manually annotate every screen recording to extract interaction patterns, error histories, and time-use decomposition.

3.3 Roadmap for the Deep Dive

First, the task selection and design rationale — why Trio, why these specific tasks, and how the tasks parallel real on-the-job learning — since the entire experiment depends on the tasks genuinely requiring new skill acquisition rather than mere Python fluency.
Second, the quiz design and psychometric validation process — how the evaluation instrument was built, tested across five versions and four pilot studies, and refined to eliminate local item dependence and syntax confounds — because the quiz is the paper's primary dependent variable and its validity determines whether the central claim about impaired skill formation is credible.
Third, the four-stage pilot study progression — the specific problems encountered (non-compliance, local item dependence, syntax barriers) and the design changes each triggered — because understanding how the final protocol emerged from iterative failure is essential for evaluating its robustness and replicability.
Fourth, the participant recruitment, screening, and balancing mechanism — the inclusion criteria, the pre-screening instrument, the stratified assignment procedure, and the demographic profile of the final sample — because the paper's claims about professional skill formation depend on the sample representing real developers facing real learning challenges.
Fifth, the treatment and control condition designs — the exact interfaces, instructions, pledges, and platform constraints for each condition — because the causal identification depends on the integrity of the manipulation.
Sixth, the qualitative annotation methodology — the event taxonomy, the coding procedure, and how interaction patterns were derived from raw screen recordings — because the paper's most actionable findings (the six personas) emerge from this analysis.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an experimental causal inference paper whose core idea is that AI assistance during a learning task impairs skill formation, but that certain interaction patterns — involving active cognitive engagement rather than passive delegation — can preserve learning outcomes.

Task Selection and Design: Why Trio and What the Tasks Actually Required

The paper's task design solves a specific measurement challenge: to study skill formation, you need participants to genuinely acquire new capabilities during the experiment. If the task only required knowledge participants already possessed, any measured "learning" differences between conditions would reflect something other than skill acquisition — perhaps motivation, fatigue, or interface familiarity. The authors needed a task that was simultaneously novel enough to require learning, feasible enough to learn within a one-hour experiment, and conceptually rich enough that the learning could be meaningfully assessed.

Why Trio specifically. The authors selected the Python Trio library, which provides asynchronous concurrency and I/O processing, for four explicit reasons documented in Section 4.1. First, Trio is less well-known than asyncio — the authors verified this through StackOverflow question counts — ensuring that participants would not have prior exposure (which the screening survey further verified by excluding anyone who had used Trio before). Second, Trio involves conceptual novelty beyond Python syntax, specifically the concept of structured concurrency — the library's core design philosophy that groups of concurrent tasks should be organized into hierarchical "nurseries" where the lifetime of child tasks is strictly bounded by the lifetime of their parent. This conceptual depth means that learning Trio is not just memorizing API calls; it requires understanding a new mental model for program execution. Third, Trio is explicitly designed to be easy to use — its documentation emphasizes approachability — making it feasible to teach within a one-hour experimental session. Fourth, asynchronous programming is a genuine professional skill that software engineers routinely learn on the job when working with large-scale data or distributed systems, giving the experiment ecological validity.

The two main tasks. After pilot testing five candidate tasks, the authors settled on two problems for the main study, each designed to take 10–20 minutes during initial testing. The decision to use two tasks rather than all five was driven by pilot study observations: using five tasks created a confound where AI-condition participants would see more Trio concepts simply because they completed more tasks within the 35-minute time limit, making it impossible to separate the quality of learning from the quantity of exposure (Section 5.1, Pilot Study C). The two-task design isolates the learning that occurs during task completion itself, holding the maximum possible concept exposure constant across conditions.

Task 1 requires implementing a timer that prints every passing second while other functions run concurrently. This introduces the core Trio concepts of nurseries (the context manager that manages groups of concurrent tasks), starting tasks (using nursery.start_soon() to spawn concurrent coroutines), and running functions concurrently (the fundamental departure from sequential execution). The problem description includes starter code and brief descriptions of these Trio concepts, simulating the experience of learning a new library through a self-guided tutorial — which the authors explicitly analogize to "software engineers' on-boarding materials" where "there is often a description of how to use an internal library and small tasks to build skills with the new library" (Section 4.1).

Task 2 requires implementing a record retrieval function that handles missing record errors using Trio's error handling mechanisms and memory channels (dictionaries, lists, or other shared data structures used to collect results from concurrent tasks, since start_soon does not return values directly). This introduces the concepts of error propagation in nurseries (how exceptions in child tasks affect parent tasks and sibling tasks), catching and handling MultiError (Trio's mechanism for aggregating multiple exceptions from concurrent children), and using memory channels to store results (the pattern for collecting output from tasks that don't return values).

What makes these tasks genuinely require learning. The tasks are not solvable through general Python knowledge alone. A developer who has never used Trio cannot simply write concurrent code using their existing mental model of threading or multiprocessing, because Trio's structured concurrency model imposes specific constraints: all concurrent tasks must be spawned within a nursery context manager, coroutines must be awaited (not called directly), and error handling follows a deterministic propagation pattern through the nursery hierarchy. The task instructions provide examples and conceptual overviews, requiring participants to read, understand, and apply this new information — exactly the skill acquisition process the paper aims to study. The starter code provides scaffolding (imports, function signatures, and structural comments) to reduce syntax friction and focus the learning on Trio-specific concepts.

The AI assistant configuration. The treatment condition's AI assistant is powered by GPT-4o and is prompted to behave as "an intelligent coding assistant" (Section 4.1). Critically, the assistant has access to participants' current version of the code — it can see what the participant has written so far — and "can produce the full, correct code for both tasks directly when prompted." This design choice is important: the AI assistant is not constrained to only provide hints or explanations; it is fully capable of solving the tasks autonomously. This means any learning differences between conditions cannot be attributed to the AI being insufficiently helpful — the AI can produce perfect solutions on demand. Instead, the differences arise from how participants choose to interact with this fully capable assistant, which is precisely what the qualitative analysis investigates.

Design rationale for the interface. The coding platform (an online interview platform with integrated code editor, terminal, and AI chat panel) was chosen to provide a controlled environment where all participant actions — keystrokes, AI queries, code runs, errors — could be recorded and later analyzed. The chat-based interface (as opposed to an autocomplete or agentic interface) was a deliberate choice: the authors argue in Section 7.1 that this represents a "lower bound for cognitive offloading since agentic AI coding tools would require even less human participation." In other words, if skill formation suffers even with a chat interface that requires users to compose queries and evaluate responses, the effect would likely be stronger with more automated tools that require less cognitive engagement. This positions the paper's findings as conservative relative to the trajectory of AI coding tool development.

Quiz Design and Psychometric Validation

The quiz is the paper's primary instrument for measuring skill formation, and its validity is critical to the central claim that AI impairs learning. The authors invested substantial effort — across five quiz versions and four pilot studies — in constructing a reliable, discriminative, and conceptually valid assessment. The design process is documented across Sections 4.2, 5.1, and Appendix C.

Question type taxonomy. The quiz operationalizes "skill formation" through three of the four assessment types identified in Cheng et al.'s (2022) meta-analysis of computer science education evaluations (Section 4.2):

Debugging questions — participants must identify and diagnose errors in provided code snippets. The authors justify this focus explicitly through the supervision argument: "This skill is crucial for detecting when AI-generated code is incorrect and understanding why it fails." Debugging questions present code with specific Trio-related bugs (for example, a nursery opened incorrectly, or a coroutine called without await) and ask participants to identify what is wrong and why.
Code reading questions — participants must read and comprehend what a provided code snippet does. The authors connect this to verification: "This skill enables humans to understand and verify AI-written code before deployment." Code reading questions present correct Trio code and ask participants to predict output, trace execution order, or explain the behavior of concurrent tasks.
Conceptual understanding questions — participants must demonstrate understanding of core principles behind Trio, rather than surface-level syntax. The authors argue this is critical for assessing "whether AI-generated code uses appropriate design patterns that adheres to how the library should be used." Conceptual questions probe understanding of nurseries, structured concurrency, error propagation, and the async/await model at a principled level.

What was deliberately excluded. The quiz does not include code writing questions. The authors provide a specific rationale: "We exclude code writing questions to reduce the impact of syntax errors in our evaluation; these errors can be easily corrected with an AI query or web search." This is a thoughtful design choice. If the quiz included code writing, participants might lose points for syntax mistakes that don't reflect genuine conceptual misunderstanding — and since AI tools can trivially fix syntax errors in production, testing syntax recall would measure an increasingly irrelevant skill. By focusing on debugging, code reading, and conceptual understanding, the quiz targets skills that remain valuable even in an AI-augmented workflow.

The 7 core concepts tested. The quiz covers seven Trio concepts that span both tasks (Section 4.2 and Appendix C.1):

Async and await keywords — when await is required, what happens when it's used or omitted, and the execution semantics of async functions.
Starting Trio functions — how to spawn tasks using nurseries, how spawned tasks with different durations behave concurrently, and the start_soon mechanism.
Error handling in Trio — how errors propagate in nurseries, what happens to parent and sibling tasks when a child raises an exception, and how to catch exceptions from concurrent children.
Coroutines — the distinction between coroutine objects and async functions, how to debug "coroutine was never awaited" warnings, and the execution model underlying async/await.
Memory channels using Trio — the pattern of using dictionaries, lists, or other shared data structures to collect results from tasks spawned with start_soon, since start_soon does not return values directly.
Opening and closing a Trio nursery — the syntax and semantics of asynchronous context managers, the async with pattern, and common mistakes in nursery setup.
Sequential vs. concurrent execution — understanding when tasks run concurrently versus sequentially, predicting task interleaving, and reasoning about execution order in Trio programs.

Psychometric validation process. The quiz underwent five iterations tested across four pilot studies (Table 2, Section 5.1). The authors used item response theory principles to validate the instrument:

Item discrimination: They ensured "all questions are sufficiently correlated with the overall quiz score" — meaning each question differentiates between high- and low-performing participants, contributing signal rather than noise.
Difficulty calibration: They verified "each question has an appropriate average score" — avoiding ceiling effects (questions everyone gets right, which provide no discriminating power) and floor effects (questions no one can answer, which frustrate participants without measuring learning differences).
Local item dependence elimination: Pilot Study C revealed a critical flaw: "participants would compare questions and identify answers based on code snippets provided in other questions." For example, a debugging question might contain a code snippet that reveals the answer to a conceptual question elsewhere in the quiz. To address this, the authors "split the quiz into several different pages, where the questions on each page did not provide hints for other questions" and verified through screen recordings in Pilot Study D that this eliminated the cross-question information leakage.

The final instrument contains 14 questions worth 27 total points, with each question belonging to exactly one task (Task 1 or Task 2) and exactly one question type (Conceptual, Debugging, or Code Reading) — enabling the per-task and per-skill-type decomposition shown in Figure 8. The grading rubric was pre-registered before running the main study.

The paper's experimental protocol emerged through a series of failures documented in Table 2 of Section 5.1. Understanding this progression is essential because each pilot study identified a specific threat to validity, and the final protocol's features are direct responses to these threats.

Pilot Study A (Platform P1, n=39, 5 tasks). The primary problem was non-compliance: 35% of participants in the no-AI condition used AI assistance anyway. The authors detected this through "coding platform transcripts of when users copied the instructions or pasted code into the editor" — participants would copy the task instructions, paste them into an external AI tool (like ChatGPT in another browser tab), and then paste the AI's response back into the coding platform. This is a fundamental threat for any study attempting to compare AI vs. no-AI conditions: how do you actually prevent the control group from using AI when AI tools are trivially accessible?

Pilot Study B (Platform P1, n=107, 5 tasks). The protocol added "more explicit instructions" and "strict no AI requirements," but non-compliance persisted: "25% of participants used AI" in the control group. The platform (P1) did not support screen recording, so the authors couldn't verify what participants were doing in real time. This revealed that instructions alone — no matter how explicit — are insufficient to prevent AI use in the control condition when participants are working remotely and have unrestricted internet access.

Pilot Study C (Platform P2, n=20, 5 tasks). The switch to Platform P2, which supported screen recordings, solved the compliance problem: "Using screen recordings of participant progress, we verified that participants did not use AI in the control group nor for the quiz." Screen recordings provide a verifiable audit trail — even if a participant were to use AI in another window, the recording would show them switching away from the platform and returning with code they didn't type.

However, Pilot C revealed a new problem: local item dependence in the quiz. Screen recordings showed "participants scrolling back and forth between questions to guess the correct answer." Participants were using information from one question to answer another — for example, a code snippet in Question 7 might reveal the correct answer to Question 3, even though the questions were designed to test independent concepts. This inflates apparent quiz performance (participants can get answers "right" without understanding the concept) and reduces the instrument's ability to discriminate genuine knowledge from test-taking strategy.

Pilot C also revealed a confound in task count: the AI group could complete more than two tasks within the 35-minute limit because AI assistance accelerated them, meaning they were exposed to more Trio concepts. This created a confound where AI-condition participants might score higher on the quiz not because they learned more per task, but because they simply saw more concepts. To fix this, the authors "reduced the total number of tasks from five to two" for the main study, ensuring both conditions had equal maximum concept exposure.

Pilot Study D (Platform P2, n=20, 2 tasks). With the task count reduced and local item dependence addressed by paginating the quiz, Pilot D tested the final protocol structure. It revealed a significant treatment effect — AI users were faster (Cohen's d=1.11, p=0.03) but scored worse on the quiz (Cohen's d=1.7, p=0.003) — but also revealed a barrier to task completion unrelated to Trio: participants in the control group "struggled with Python syntax that was unrelated to Trio, such as try/except blocks and string formatting." The task completion rate was only 60% in the control group (compared to 90% in the AI group), meaning many control participants simply couldn't finish the tasks within the time limit due to Python syntax issues that had nothing to do with the library being studied.

This is a critical validity concern: if the control group can't finish the tasks, any learning differences might reflect differing amounts of practice with Trio (the AI group gets more exposure because they complete more tasks) rather than the quality of learning during task completion. The authors' fix was to add "syntax hints about string formatting and try/except blocks for the main study" — essentially, providing a limited form of assistance to both conditions for Python mechanics so that the experimental contrast isolates Trio-specific learning.

The authors used Pilot D's effect sizes to power their main study. The observed learning effect was d=1.7, but they "assumed a conservative effect size of d = 0.85 (half of the observed learning effect) to account for the potential effect size inflation typical in pilot studies" when pre-registering their target sample size.

Participant Recruitment, Screening, and Balancing

Inclusion criteria. The paper recruited participants who satisfied five criteria (Section 4.3):

More than one year of Python experience.
Code in Python at least once a week.
Have tried AI coding assistance at least a few times.
Have never used the Trio library before.
Self-reported through a recruitment survey covering years of coding experience, years of Python experience, prior usage of Python's asyncio library, frequency of Python use, and an asynchronous programming familiarity score (a 5-question multiple-choice concept check).

These criteria serve specific experimental purposes. The first two ensure participants have baseline Python fluency — the experiment studies learning a new library, not learning to program. The third criterion eliminates a confound where treatment effects could reflect novelty of the AI interface rather than AI assistance itself — all participants are familiar with AI coding tools, so any differences are attributable to use during the task rather than unfamiliarity with the interface. The fourth criterion is essential for studying skill formation: if participants already knew Trio, the quiz would measure existing knowledge rather than learning. The fifth criterion (the pre-screening survey) enables balanced assignment — ensuring that any treatment-control differences are not driven by systematic differences in prior asynchronous programming knowledge.

Assignment and balancing. Participants were randomly assigned to treatment (AI assistant) or control (no AI) conditions. Table 1 reports the balance across covariates: years of coding experience, frequency of Python use, prior asynchronous programming knowledge (measured by a 5-question pre-task quiz scored 0–5), prior usage of Python's asyncio library, and pre-task warm-up coding time. The groups are well-balanced on all measured covariates — for example, 7+ years coding experience is 53.8% in treatment vs. 57.7% in control, and prior asyncio usage is 69.2% vs. 77.0%. The warm-up task completion time is 6.5 minutes (treatment) vs. 8 minutes (control), a non-significant difference that confirms similar baseline Python proficiency.

Sample size and statistical power. The main study recruited 52 participants (26 per condition) after sending the study to 58 crowd workers. One additional participant completed the study but was disqualified under pre-registered criteria for "leaving four blank questions on the quiz due to not realizing that there were multiple parts of the quiz and subsequently running out of time" — a procedural exclusion, not a selective one. The final sample was 52 participants.

The power analysis used the conservative d=0.85 estimate from the pilot, which with n=52 yields reasonable power to detect a large effect — consistent with the significant result obtained (Cohen's d=0.738, p=0.010).

Participant demographics. Figure 17 (Appendix A) shows that most participants hold a bachelor's degree, are between 25 and 35 years old, and work either as freelance or professional software developers. This demographic profile is important for the paper's ecological validity claims: the participants are not students completing a classroom exercise but practicing professionals facing a task that resembles on-the-job learning. The authors emphasize this in Section 7.1, noting that "while participants in our study were professional or freelance programmers, there was not the same incentive to learn the library as if it were required for their actual job" — acknowledging a limitation while maintaining that the sample is more representative of the target population than a student sample would be.

Compensation and incentive structure. Participants were paid a flat rate of $150 USD for a task taking an average of 58.5 minutes (maximum 1 hour 15 minutes). The flat-rate compensation is deliberate: it avoids conflating speed incentives with learning incentives. If participants were paid by completed task, the AI condition would have a strong financial incentive to delegate everything and move fast, artificially inflating the productivity difference. If participants were paid by quiz score, the control condition might have a stronger incentive to learn, artificially inflating the learning difference. The flat rate makes the experimental contrast about what happens naturally when people use AI to complete a learning task, rather than about how payment structures interact with AI use.

Ethical review. The protocol was "reviewed and approved by internal reviewers at Anthropic" (Appendix A.1). Participants provided informed consent during pre-screening, were given the right to withdraw at any time without penalty, and were guaranteed compensation regardless of task completion or quiz performance. Data was anonymized — only the crowd-worker platform could link identities to payments, and the researchers removed platform identifiers before analysis.

Treatment and Control Condition Designs

The experimental manipulation is a between-subjects design with careful attention to platform constraints, instruction framing, and compliance verification.

The warm-up task (all participants, no AI). Every participant begins with the same task: write Python code to add a border around a list of strings. This problem uses no asynchronous concepts and takes approximately 4 minutes on average on the platform. No participants have access to AI during this stage. The warm-up serves three purposes:

Calibration: it provides a measure of baseline Python fluency that can be used as a covariate in analysis (the paper reports results both with and without warm-up time as a covariate, Section 5.2.2).
Platform familiarization: participants learn the coding platform interface (editor, terminal, run button) before the main task, reducing confounds from interface learning.
Attention and engagement verification: the warm-up serves as a basic screening — participants who cannot complete a simple Python task are unlikely to be able to attempt the Trio tasks meaningfully.

The main Trio stage (35 minutes, treatment varies). Both conditions receive identical task descriptions, starter code, and Trio concept introductions for the two tasks. Both have access to web search (though the platform restricts tab-switching, making web search somewhat cumbersome). Both are instructed to "complete the task as fast as they could."

The treatment difference is solely the availability of the AI assistant. In the treatment (AI) condition, the platform includes a chat panel connected to GPT-4o, prompted as an intelligent coding assistant with access to the participant's current code. The interface includes "a nudge to use the AI assistant on the left tool pane" (Figure 26 in Appendix D). In the control (no AI) condition, no chat panel is available. Control participants are also required to take an explicit pledge (Figure 21, Appendix D): "I understand that I must NOT use any AI tools including chatbots (such as ChatGPT, Claude, Copilot, Gemini, etc) to help me with the coding task. I will not type any part of this task into another tab, window, or application." The treatment group takes a different pledge (Figure 22) that encourages AI use.

Compliance enforcement. The paper's experience with Pilots A and B demonstrated that pledges alone are insufficient. The critical enforcement mechanism in the main study is screen recording: every participant uploaded a screen recording of their entire session (warm-up, main task, and quiz). The authors manually watched all 51 available recordings (one treatment participant's recording was unavailable) to verify that control participants did not use AI and that treatment participants' AI interactions matched the platform transcripts. This is labor-intensive but essential for internal validity — without it, the study would be measuring the effect of being assigned to the AI condition rather than the effect of actually using AI, which are different if non-compliance is high.

The quiz stage (all participants, no AI). After the 35-minute task period (or when they finish early), all participants complete the quiz without AI access. The quiz is administered as a Google Form, and the screen recordings verify that no participants used AI during the quiz. The post-quiz survey collects demographic information, self-reported enjoyment, self-reported learning, and self-reported task difficulty (on 7-point scales), plus open-ended feedback.

Timing constraints. The total session has a maximum duration of 1 hour 15 minutes, with an average completion time of 58.5 minutes. The 35-minute task time limit was set based on pilot testing: it allowed most participants in both conditions to complete both tasks (in the main study, 22 of 26 control participants and all 26 treatment participants completed Task 2), while still creating time pressure that discourages excessively elaborate AI interactions (the authors wanted to observe naturalistic behavior under moderate time pressure, not open-ended exploration).

Qualitative Annotation Methodology

The qualitative analysis in Section 6 is not merely illustrative — it is the mechanism by which the paper explains why the quantitative results occurred and identifies what distinguishes effective from ineffective AI use. This section describes the annotation procedure, the event taxonomy, and how the six interaction personas were derived.

Data source. 51 of 52 participants uploaded screen recordings covering the warm-up, main coding task, and quiz. The authors watched the main coding task portion of every recording for all participants (25 AI condition, 25 no AI condition — one AI participant's recording was missing, hence 25 rather than 26). This means every qualitative claim about interaction patterns is grounded in direct observation of every participant's behavior, not sampled or inferred from aggregate metrics.

Event taxonomy. The authors coded eight event types with timestamps (Table 5, Appendix B.1):

Task Start — when a user opens each task, enabling per-task time decomposition.
AI Interaction — when a user starts typing into the AI chat window, including a description of what they're asking and how they're composing the query.
AI Query — when the AI assistant produces a response, labeled with the query type (explanation, generation, debugging, capabilities question, or appreciation).
Websearch — when a user queries a search engine, including the search query.
Paste (Direct) — when a user pastes AI output directly into their code editor.
Code Copying — when a user manually types code while referencing AI output (as opposed to pasting).
Error — when code produces an error on execution, including the full error message.
Interface Error — when the development environment or AI assistant itself produces an error (distinct from code errors).
Task Completion — when correct output is achieved, enabling per-task completion time measurement.
Task Submission — when the user submits the final code.

This taxonomy enables several downstream analyses: error frequency by condition and error type (Figure 15), active coding time versus AI interaction time (Figure 16), query type distributions (Table 3), and adoption method comparisons (Figure 13).

Query type classification. AI queries were categorized into five types with examples provided in Table 3:

Explanation queries (q=79) — the most common type. Participants asked about Trio concepts ("can trio.sleep use partial seconds?"), asynchronous operations ("Can you remind me what the different trio async operations are?"), and high-level understanding ("can you give me a really brief overview of the general idea behind all of this?"). 21 of 25 treatment participants asked at least one explanation question.
Generation queries (q=51) — requests for code. These ranged from entire task solutions ("given this instruction to trio, can you implement the missing bits of main.py?") to specific functions ("implement delayed_hello()"). Only 16 of 25 participants used AI to generate code, and 4 participants only asked generation queries with no other types — a pattern associated with the lowest quiz scores.
Debugging queries (q=9) — requests to fix errors or verify correctness. These included pasted error messages ("Traceback (most recent call last)...") and confirmation requests ("Does that look right?"). The paper notes that a higher fraction of debugging queries correlates with slower completion times and lower quiz scores (Figure 19), suggesting that "relying on AI for debugging (e.g., repeatedly asking AI to check and fix things without understanding) when learning a new task is correlated with less learning."
Capabilities questions (q=4) — meta-questions about what the AI can do ("Can you see the current question?", "Are you aware of how trio works?").
Appreciation (q=4) — expressions of gratitude after task completion ("Great job, we got the expected output on the first try.").

Some queries received multiple labels (11 queries had two labels).

Active coding time measurement. The authors distinguish raw "total task time" (the primary quantitative productivity metric) from "active coding time" — the time actually spent typing code, as opposed to reading instructions, composing AI queries, reading AI responses, debugging errors, or thinking. Active coding time was measured from screen recordings by identifying periods when the participant was actively editing code in the editor. This decomposition (Figure 16) reveals that AI shifts how participants spend their time: treatment participants spend less time actively coding and more time reading, composing queries, and waiting for AI responses. This finding connects to the paper's central argument about cognitive offloading — the productivity stagnation in the AI condition occurs because time saved on coding is partially or fully offset by time spent interacting with the AI.

Deriving the six interaction personas. The six interaction patterns (Figure 11) were not pre-registered; they emerged from the qualitative analysis as the authors identified clusters of behavior along four axes: query types, number of queries, queries per task, and active time. The derivation process was:

Annotate every screen recording with the event taxonomy, producing a timestamped behavioral transcript for each participant.
Identify common patterns across participants in how they used AI — some participants only asked conceptual questions, others progressively delegated more to AI, others used AI for debugging in tight loops.
Group participants into clusters based on similar behavioral profiles.
Name and characterize each cluster with a descriptive label and compute average task completion times and quiz scores for each group.
Validate that the clusters have meaningfully different outcomes — the gap between low-scoring clusters (24–39% average quiz score) and high-scoring clusters (65–86% average quiz score) is stark, confirming that the behavioral patterns capture real differences in learning effectiveness.

The six patterns divide naturally into two outcome categories:

Low-scoring patterns (quiz scores below 40%):

AI Delegation (n=4) — participants who wholly relied on AI to write code, completed tasks fastest, encountered few or no errors, and scored poorly on the quiz because they never engaged with Trio concepts directly.
Progressive AI Reliance (n=4) — participants who started by asking 1–2 questions and then progressively delegated all code writing to AI, scoring poorly "largely due to not mastering any of the concepts in the second task" — they lost engagement over time.
Iterative AI Debugging (n=4) — participants who used AI to debug or verify code in tight feedback loops (many queries, repeated checking), scoring poorly and completing tasks slowly because they "relied on the assistant to solve problems, rather than clarifying their own understanding."

High-scoring patterns (quiz scores above 65%):

Generation-Then-Comprehension (n=2) — participants who first generated code (like the Delegation group) but then "asked the AI assistant follow-up questions to improve understanding," converting the AI output into a learning opportunity through active interrogation.
Hybrid Code-Explanation (n=3) — participants who "composed hybrid queries in which they asked for code generation along with explanations of the generated code," investing time in reading and understanding the explanations alongside the code.
Conceptual Inquiry (n=7) — participants who "only asked conceptual questions and relied on their improved understanding to complete the task," encountering many errors but resolving them independently. This was the fastest high-scoring pattern and the second-fastest overall.

The persona analysis transforms the paper's contribution from a simple "AI hurts learning" finding into a more nuanced "certain ways of using AI destroy learning while other ways preserve it" framework. This is the actionable insight for practitioners: the goal is not to avoid AI entirely, but to maintain cognitive engagement during AI-assisted learning.

4. Key Insights and Innovations

Innovation 1: The Product-Versus-Process Diagnostic — Reframing AI Evaluation Around Skill Formation, Not Just Productivity

The paper's most fundamental conceptual move is its insistence that evaluating AI tools solely by their output — lines of code written, tasks completed per hour, pull requests merged — is an incomplete, potentially dangerous framing. This is not merely a methodological preference; it is a redefinition of the evaluation target. Prior studies on AI for software engineering (Peng et al., 2023; Cui et al., 2024) measured productivity gains and found them substantial, particularly for junior developers. The dominant narrative that emerged from this work was optimistic: AI tools democratize coding, level the playing field, and accelerate novice workers. The paper does not dispute these productivity findings, but argues they answer only half the question — and arguably the less important half for long-term workforce development.

The conceptual innovation is the product-versus-process diagnostic: the distinction between what a worker produces during AI-assisted work and what they retain afterward as durable expertise. This framing is crystallized in the paper's explicit statement in Section 1:

"Although most studies have focused on the end product of AI assistance (e.g., lines of code written, quality of ideas proposed), an equally important, if not more crucial question is how process of receiving AI assistance impacts workers."

This is not a small refinement of existing evaluation frameworks — it is a fundamental reframing that changes what counts as a successful AI deployment. Under the product-only view, a tool that makes junior developers twice as fast is an unambiguous success. Under the product-and-process view, that same tool might be a net negative if it leaves those developers unable to debug, verify, or extend the code they produced — and unable to supervise more capable AI systems in the future. The paper operationalizes this reframing by measuring both productivity and skill formation simultaneously (Figure 6), and the discordance between the two metrics — no significant productivity gain, yet significant skill loss — demonstrates that they capture genuinely distinct phenomena.

The significance of this reframing extends beyond software engineering. The paper explicitly connects it to the "scalable oversight" problem (Bowman et al., 2022): as AI systems become more capable, humans remain legally and ethically responsible for their outputs. If AI tools erode the skills needed for oversight during the process of using them, we create a self-reinforcing dependency where workers can neither function without AI nor evaluate its outputs critically. This is a new diagnostic concept — not just that AI causes overreliance (which the overreliance literature already documents: Buçinca et al., 2021; Vasconcelos et al., 2023), but that AI-assisted task completion can prevent the formation of the very expertise needed to avoid overreliance in the first place. The overreliance literature focuses on decision-time behavior; this paper shows that the problem starts earlier, during skill acquisition.

The product-versus-process distinction also explains why prior observational work was insufficient. Surveys finding correlations between AI use and reduced critical thinking (Gerlich, 2025; Lee et al., 2025) cannot establish whether AI causes skill erosion or whether workers with weaker skills self-select into heavier AI use. The paper's randomized design, combined with the process-focused measurement, converts a correlation the field already suspected into a causal claim with clear direction: AI assistance during learning tasks impairs skill formation, and the impairment is substantial (Cohen's d=0.738, Figure 6).

Innovation 2: The Interaction Persona Taxonomy — Identifying How AI Is Used as the Mediating Variable for Skill Preservation

The paper's second major contribution is the finding that AI's effect on learning is not uniform but is mediated by interaction patterns — and the specific patterns that preserve learning are identifiable and teachable. This moves the conversation from "should we use AI for learning?" (a binary, largely unproductive question) to "how should we use AI for learning?" (a practical, actionable question with demonstrated answer patterns).

Prior work on AI usage patterns in coding education (Poitras et al., 2024; Wang et al., 2024; Prasad et al., 2023) had described what students and developers do with AI tools — write code, fix errors, ask questions — but these were descriptive taxonomies without outcome correlates. They could tell you that some students ask for explanations while others ask for code generation, but not which patterns were associated with better or worse learning. The field's implicit assumption was that any AI assistance during learning was roughly equivalent or that more sophisticated prompting would naturally lead to better outcomes.

The paper's qualitative analysis (Section 6) breaks this assumption by tying interaction patterns to measured skill outcomes. The six personas (Figure 11) are not merely colorful labels — they are empirically grounded clusters with starkly divergent quiz scores: low-scoring patterns (AI Delegation, Progressive AI Reliance, Iterative AI Debugging) average 24–39%, while high-scoring patterns (Generation-Then-Comprehension, Hybrid Code-Explanation, Conceptual Inquiry) average 65–86%. This 40+ percentage point gap within the AI-using group — larger than the overall treatment-control difference — demonstrates that how AI is used matters more than whether it is used.

The intellectual contribution here is not the taxonomy itself, but the diagnostic principle it reveals: cognitive engagement is the mediating variable. The three high-scoring patterns share a common feature — they require the participant to actively process, interrogate, or apply AI outputs rather than passively accepting them. Conceptual Inquiry users never ask for code; they ask for understanding and write code themselves. Generation-Then-Comprehension users treat AI output as a starting point for follow-up questioning, not a final answer. Hybrid Code-Explanation users demand explanations alongside code. In all three cases, the AI is a tool for thinking, not a substitute for thinking — and this distinction, not AI access per se, determines whether learning occurs.

This insight is fundamental rather than incremental because it changes the prescriptive landscape. If the paper had only found that AI impairs learning (Innovation 1), the natural policy response would be restriction: ban AI during training, delay AI adoption until after expertise is established, or wall off learning tasks from AI tools. The persona analysis complicates this picture: restriction is one path, but redesigning AI interaction to enforce cognitive engagement is another. The paper notes that major LLM services already provide "learning modes" (ChatGPT Study Mode, Claude Code Learning/Explanatory mode), and the persona taxonomy provides an empirical foundation for designing such modes — they should encourage Conceptual Inquiry and Hybrid Code-Explanation patterns while discouraging pure Delegation.

The persona analysis also explains the paper's null productivity result in a way that the aggregate statistics alone cannot. The overall treatment-control difference in task completion time is non-significant (Figure 6), but Figure 13 reveals why: only the 9 participants who directly pasted AI code finished substantially faster than the control group. The 9 participants who manually typed AI-generated code and the 4 who used a hybrid approach took roughly as long as the no-AI group. The 4 participants who mostly wrote their own code while asking only clarification questions were actually quite fast. This decomposition shows that the aggregate null result masks heterogeneous treatment effects — AI can speed up task completion, but only for a specific interaction pattern (direct pasting), and that pattern is also associated with the worst learning outcomes. Productivity gains and skill preservation appear to be in tension, not because AI is inherently speed-versus-learning tradeoff, but because the fastest interaction patterns are also the most cognitively passive.

Innovation 3: Encountering and Resolving Errors as the Mechanism of Skill Formation — An Empirical Test of Experiential Learning Theory in the AI Era

The paper's third distinctive contribution is its empirical documentation of error encounters as a causal mechanism for skill acquisition, and the demonstration that AI assistance short-circuits this mechanism by reducing both the frequency of errors and the cognitive engagement required to resolve them. This is not merely a finding about debugging skill — it is a mechanistic explanation for why AI impairs learning that connects the observed quiz score differences to observable differences in the learning process.

Experiential learning theory (Kolb, 2014) and Problem-Based Learning (Schmidt, 1994) have long argued that encountering and resolving difficulties is essential for deep learning — that the "struggle" phase of problem-solving is not an unfortunate inefficiency but the engine of conceptual understanding. The paper provides what is, to my knowledge, the first direct experimental evidence quantifying this mechanism in the context of AI-assisted learning. The evidence chain is specific and causal:

First, the AI group encountered dramatically fewer errors: the median participant in the treatment condition encountered only 1 error across both tasks, compared to 3 errors for the median control participant (Table 4). Ten of the 12 participants who completed both tasks without any errors were in the AI condition.

Second, and more importantly, the types of errors differed systematically. Figure 15 shows that the most common errors in both conditions were non-Trio-specific (NameError, AttributeError — typical typos). However, the control group encountered substantially more Trio-specific errors: RuntimeWarning (indicating a coroutine was never awaited) and TypeError (indicating a coroutine object was passed where an async function was expected). These errors force engagement with core Trio concepts — the distinction between coroutine objects and async functions, the requirement to await coroutines, the structured concurrency execution model. The control group encountered these concept-specific errors; the AI group largely bypassed them because the AI assistant either generated correct code directly or fixed errors without requiring the participant to understand why the fix worked.

Third, the largest quiz score gap between conditions appears on debugging questions (Figure 8) — exactly the skill category that error encounters would be expected to develop. The control group's experience of repeatedly encountering, diagnosing, and fixing Trio-specific errors during the task translated directly into superior ability to identify and explain errors in the quiz. The AI group, having delegated error resolution to the assistant, lacked this developmental experience.

This finding is significant because it operationalizes a theoretical mechanism that had previously been assumed but not empirically demonstrated in the AI context. It shows that the experiential learning cycle is not just a pedagogical philosophy — it is an empirically observable process that AI tools can disrupt by removing the "concrete experience" and "reflective observation" stages that error encounters provide. When the AI assistant fixes a RuntimeWarning by inserting the missing await keyword, the participant sees the correct code but does not experience the diagnostic reasoning that connects the error message to the underlying concept. The learning is bypassed, not accelerated.

The practical implication is counterintuitive and important: errors are not bugs in the learning process; they are features. A well-designed AI learning tool should not eliminate errors but should help learners engage with errors — providing explanations, asking diagnostic questions, or offering hints rather than solutions. The paper's finding that Iterative AI Debugging (repeatedly asking AI to fix errors without understanding) produced poor quiz scores, while Conceptual Inquiry (asking only conceptual questions and resolving errors independently) produced high scores, reinforces this point: it is not the presence of errors that matters but how the learner engages with them.

Innovation 4: The Exoskeleton Reversal — Demonstrating That AI Assistance Impairs Skill Formation in the Same Population That Benefits Most from It

The paper's fourth contribution is a diagnostic insight about distributional effects that inverts the standard narrative about AI and expertise. The dominant finding in the AI productivity literature — replicated across call centers, consulting, legal analysis, writing, and software engineering (Brynjolfsson et al., 2025; Dell'Acqua et al., 2023; Noy and Zhang, 2023; Peng et al., 2023; Cui et al., 2024) — is that less experienced, lower-skilled workers benefit most from AI assistance. This finding has been celebrated as evidence that AI democratizes expertise, closing performance gaps between novices and experts.

The paper does not challenge this finding — but it does challenge the inference that AI therefore helps novices develop into experts. The diagnostic move is to point out that productivity during a study and skill development across a career are different dependent variables, and that the very mechanism producing the productivity gain (AI compensating for missing knowledge) may simultaneously be the mechanism preventing that knowledge from being acquired. The paper's results support this concern: Figure 7 shows that the control group outscores the AI group on the quiz across every level of coding experience (1–3 years, 4–6 years, 7+ years), meaning the skill impairment is not confined to novices — it affects experienced developers learning a new library as well.

The conceptual contribution is what I call the exoskeleton reversal: Wiles et al. (2024) found that AI acts as an "exoskeleton" for non-technical workers — providing capabilities that disappear when the AI is removed. The paper extends this finding by showing that the exoskeleton not only fails to transfer skills but actively prevents their acquisition — and this effect is present even for technical professionals who have substantial domain expertise. The exoskeleton doesn't just fail to become part of the wearer; it atrophies the muscles it was meant to support.

This insight has significant implications for how organizations should think about AI deployment in learning-intensive roles. If the standard finding is that AI helps novices most, a naive manager might assign AI tools preferentially to junior developers — maximizing short-term productivity while inadvertently minimizing long-term skill development. The paper's finding suggests the opposite strategy may be more prudent: give AI tools to experienced developers (whose foundational skills are already established and less vulnerable to atrophy) while protecting junior developers from AI during learning-intensive tasks, or — following the persona taxonomy — giving them AI with constraints that enforce cognitive engagement (explanation modes, conceptual-only querying, generation-then-comprehension workflows).

The paper explicitly frames this concern in Section 7:

"Given time constraints and organizational pressures, junior developers or other professionals may rely on AI to complete tasks as fast as possible at the cost of real skill development."

This is not a hypothetical — it is the natural outcome of organizations optimizing for the metrics they measure (task completion speed, pull request volume) rather than the outcomes they should care about (developer expertise, code quality, long-term autonomy). The paper's contribution is to make this tradeoff visible and measurable, providing the empirical foundation for organizations to design AI deployment policies that balance short-term productivity with long-term skill formation.

5. Experimental Analysis

Evaluation Methodology

Dataset. The MATH benchmark (Hendrycks et al., 2021), consisting of high-school competition-level mathematics problems. The paper uses the specific split from Lightman et al. (2022): 12,000 training questions and 500 test questions. The choice is deliberate (Section 4): test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences — mathematical reasoning fits this profile because it requires multi-step logical deduction rather than novel factual recall.
Base model. All experiments use PaLM 2-S* (Codey) (Anil et al., 2023). The authors argue this model is "representative of the capabilities of many contemporary LLMs" and sits in a useful regime: non-trivial performance on MATH (roughly 10–19% pass@1 depending on the prompt and sampling configuration) but far from saturation, leaving room for test-time compute to make a difference. For the FLOPs-matched comparison, a second model with approximately 14× more parameters is used as the pretraining-scaled baseline.
Metrics. The primary metric throughout is MATH test accuracy (%) — the fraction of the 500 test questions for which the selected final answer matches the ground truth. Answers are graded using the grading function released by Lightman et al. (2022) (Appendix G). When analyzing difficulty-dependent behavior, the paper reports accuracy within each of the five difficulty quintiles separately.
Baselines. The paper uses several distinct baselines, each serving a specific comparison purpose. Majority voting selects the most common final answer among N sampled solutions, requiring no learned verifier — this tests whether simple consensus can replace verifier training. ORM best-of-N weighted scores N solutions with an outcome reward model (a model trained to predict final-answer correctness) and applies best-of-N weighted selection — this tests whether per-step scoring (PRM) improves over final-answer-only scoring (ORM). PRM best-of-N weighted scores N solutions with the process reward model and applies best-of-N weighted selection — this is the primary verifier-augmented baseline against which search algorithms are compared. For the revision experiments, parallel sampling generates N independent solutions from the revision model and selects the best via verifier or majority voting — isolating the effect of sequential versus parallel allocation.
Generation budget / compute accounting. One "generation" equals one complete sampled answer from the base LLM. For beam search and best-of-N, the budget equals the number of beams or samples N. For lookahead search with k lookahead steps, the cost is N × (k+1) to account for the additional rollout computation (Section 5.3). Budgets are swept across powers of 2, typically from 2⁰ to 2⁹ (1 to 512 generations). This enables direct comparison of different methods at equal total compute expenditure, without penalizing methods that use computation differently (e.g., beam search spends compute on intermediate steps; best-of-N spends it on complete solutions).
Cross-validation / statistical protocol. To avoid contaminating strategy selection with test-set performance, the authors use two-fold cross-validation within each difficulty bin on the 500-question test set. The best-performing strategy is selected on one fold and evaluated on the other, with results averaged (Section 3.2). This prevents the compute-optimal policy from being evaluated on the same data used to select it, providing an unbiased estimate of generalization performance. The same protocol is applied for both oracle difficulty bins (computed from ground-truth correctness) and predicted difficulty bins (computed from PRM final-answer scores).

Main Quantitative Results

Search Against PRM Verifiers

The paper's first major experimental axis evaluates three search algorithms — best-of-N weighted, beam search, and lookahead search — all operating against the same PRM verifier, systematically sweeping generation budget and analyzing results by difficulty bin.

Aggregate search algorithm comparison (Figure 3, left). Across all 500 test questions with a maximum budget of 256 generations, the performance ordering depends on budget level. At low budgets (2–8 generations), beam search with M=4 significantly outperforms best-of-N weighted. The paper reports that at 4 generations, beam search (M=4) achieves roughly 27% accuracy versus roughly 16% for best-of-N weighted — an 11 percentage point gap. This demonstrates that guided search is substantially more sample-efficient than independent sampling when the budget is small.

At high budgets (64–256), the pattern reverses: beam search performance flattens and falls slightly below best-of-N weighted. Best-of-N weighted reaches approximately 38% at 512 generations; beam search (M=4) plateaus around 34%. This degradation at high budgets is the paper's primary evidence for PRM over-optimization — search finds solutions that score highly under the verifier but are actually incorrect.

Lookahead search — the most sophisticated and expensive optimizer — paradoxically performs worst overall. Both k=1 and k=3 variants generally underperform at the same generation budget because the additional rollout cost (k+1) reduces the effective number of beams explored. The 3-step lookahead variants converge to similar performance as other methods only at the highest budgets but never surpass them, providing further evidence that verifier over-optimization, not search algorithm sophistication, is the binding constraint.

Majority voting trails all verifier-based methods substantially, reaching only about 29% at 512 generations — confirming that the PRM provides genuine signal beyond simple answer consensus, though that signal degrades under aggressive optimization.

Difficulty-bin analysis for search (Figure 3, right). Breaking out beam search (M=4) versus best-of-N weighted by difficulty quintile at four budget levels (4, 16, 64, 256 generations) reveals the paper's most important pattern:

Bin 1 (easiest questions, highest base-model pass@1): Beam search accuracy decreases from roughly 78% to 77% as the budget goes from 4 to 256 generations, while best-of-N weighted increases from 68% to 88%. This is the clearest evidence of PRM over-optimization: the PRM's signal is reliable enough on easy problems that best-of-N weighted sampling benefits from more samples, but beam search amplifies residual verifier errors by aggressively optimizing against an imperfect proxy. The search process finds solutions that exploit the verifier's blind spots rather than genuinely better solutions.
Bin 2: Beam search improves modestly (roughly 14% → 32%) but best-of-N weighted improves faster (roughly 14% → 60%), maintaining a clear advantage at high budgets. The verifier signal is less reliable than in Bin 1, but still sufficiently good that independent sampling with verifier-based selection outperforms aggressive search.
Bin 3 (medium difficulty): Beam search consistently outperforms best-of-N weighted across all budgets, reaching roughly 34% vs. 23% at 256 generations. This is the regime where verifier-guided search genuinely helps — the PRM's per-step guidance navigates toward correct solutions that the model wouldn't find by random sampling alone, and the verifier signal is strong enough relative to problem difficulty that over-optimization hasn't yet set in.
Bin 4 (medium-hard): Beam search shows the strongest relative advantage, reaching roughly 17% vs. 10% for best-of-N at 256 generations. The absolute performance is low, but the relative gain from search is largest here — guided exploration is particularly valuable when correct solutions are sparse but the verifier can still recognize promising partial solutions.
Bin 5 (hardest questions, near-zero base-model pass@1): Both methods hover near 1–3% regardless of budget. No method makes meaningful progress. This is the capability ceiling: when the base model essentially never produces correct solutions, no verifier or search algorithm can find what isn't there.

Compute-optimal search (Figure 4). By selecting the best search strategy per difficulty bin at each budget level, the paper demonstrates substantial efficiency gains. At 16 generations, compute-optimal search with oracle difficulty bins achieves approximately 27% accuracy, roughly matching PRM best-of-N weighted at 64 generations — a 4× reduction in required compute for equivalent accuracy. At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (roughly 37%).

Critically, the predicted difficulty bins (using PRM scores rather than ground-truth correctness to estimate difficulty) track the oracle version closely, particularly at lower budgets. The paper states the two curves "largely overlap" (Figure 4), with the predicted version reaching approximately 37% at 256 generations. Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (around 29%), confirming that the adaptive strategy works without access to ground-truth labels and that PRM-based difficulty estimation provides a sufficient proxy for oracle difficulty.

PRM vs. ORM comparison (Figure 14, Appendix F). At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. The gap between PRM and ORM widens with the number of samples, confirming the PRM's superior scaling properties and validating the choice of step-level training over final-answer-only training, even when last-step aggregation is used at test time — the PRM training acts as beneficial representation learning.

Revision Model Results

The second major experimental axis evaluates iterative revision — where the model conditions on its own previous incorrect answers and produces improved answers sequentially — comparing sequential versus parallel allocation of a fixed generation budget.

Revision model pass@1 trajectory (Figure 6, left). The revision model starts at approximately 18.2% pass@1 at step 1 (its first attempt without revision history). Per-step accuracy improves to roughly 24–25% by steps 15–20 and remains in the 23–25% range out to 64 steps. This gradual improvement over the chain demonstrates that the model has learned a generalizable revision skill that continues to yield marginal improvements well beyond its 4-step training horizon — evidence that the offline training procedure (pairing incorrect answers with correct ones by edit distance) captured a transferable capability rather than merely memorizing training trajectories.

Sequential vs. parallel (Figure 6, right). At 64 generations (matching the budget between conditions):

Sequential + best-of-N weighted: approximately 41.5%
Parallel + best-of-N weighted: approximately 39%
Sequential + majority: approximately 38%
Parallel + majority: approximately 35%

Sequential outperforms parallel under both selection mechanisms. The verifier-based gap (roughly 2.5 percentage points) is slightly narrower than the majority-based gap (roughly 3 points), suggesting that the verifier partially compensates for the lower quality of parallel samples but that sequential revision provides genuine quality improvement in the candidates themselves, not just an artifact of the verifier seeing revision history.

Sequential-to-parallel ratio sweep (Figure 7, left). For a fixed generation budget, the paper sweeps the ratio of sequential depth to parallel breadth. At 256 generations, the optimal ratio is around 2¹ to 2³ (2:1 to 8:1 sequential-to-parallel), achieving approximately 43–44% accuracy. The extremes perform worse: fully parallel (leftmost point) yields approximately 40%, and fully sequential (rightmost point) yields approximately 42%. This inverted-U shape demonstrates that neither pure breadth nor pure depth is optimal — some combination of exploring diverse approaches (parallel) and refining within each approach (sequential) is best.

At lower budgets (8–32 generations), the curve shifts: fully sequential is optimal, with the curves monotonically increasing with the sequential-to-parallel ratio. When the total budget is small, it's better to do deep refinement on a single chain than shallow exploration across many chains — the initial attempts are likely good enough that refinement dominates over diversity.

Difficulty-dependent ratio (Figure 7, right). Fixed at 128 generations, the paper breaks out the sequential-to-parallel tradeoff by difficulty quintile:

Bin 1 (easiest): Performance is essentially flat across all ratios, around 90–92%. Easy questions are robust to allocation strategy — the model can solve them regardless of how the budget is deployed.
Bin 2: Slight advantage for higher sequential ratios, approximately 63% at fully sequential vs. 58% at fully parallel. Refinement dominates over exploration.
Bin 3: A clear optimal ratio emerges at moderate sequential-to-parallel values (around 2¹ to 2³), reaching approximately 42% vs. 35% at the extremes. The 7 percentage point gap between optimal and suboptimal ratios demonstrates that getting the allocation right matters substantially.
Bin 4 (medium-hard): Similar pattern, with the peak at a moderate ratio achieving roughly 18% vs. 14% at fully parallel. The absolute improvement is smaller but still meaningful.
Bin 5 (hardest): All ratios produce roughly 2–3% accuracy. When the base model fundamentally cannot solve the problem, no allocation strategy helps.

This difficulty-dependent pattern mirrors the search findings: easy problems benefit from exploitation (sequential refinement), harder problems benefit from a balance of exploration and exploitation (mixed ratio), and the hardest problems see no benefit from any strategy.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty bin at each budget level:

At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations — a 4× improvement in compute efficiency.
At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and 37% for parallel-only.
Compute-optimal with predicted bins performs slightly below oracle at high budgets (approximately 41% at 256 generations) but still substantially outperforms the parallel baseline.
The parallel baseline appears to plateau around 36–37% at high budgets, while compute-optimal scaling continues to improve, suggesting that the gains from adaptive allocation compound at higher budgets and that parallel-only strategies hit a ceiling that adaptive strategies can surpass.

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute

Section 7 asks: given a fixed total FLOPs budget, is it better to train a larger model or to keep the smaller model and spend the extra FLOPs on inference-time computation? The comparison is between PaLM 2-S* with compute-optimal test-time scaling and a model with approximately 14× more parameters using greedy decoding with no extra test-time compute.

FLOP accounting. The paper uses standard approximations from the scaling laws literature: pretraining FLOPs X = 6ND_pretrain and inference FLOPs Y = 2ND_inference, where N is the number of parameters. Scaling parameters by a factor of M multiplies both X and Y by M. To match the total FLOPs of the M×-larger model using the smaller model with additional test-time compute, the smaller model's inference budget multiplier depends on the ratio R = D_inference / D_pretrain. Three values of R are tested: 0.16 (R ≪ 1, few inference tokens relative to pretraining — the smaller model gets a large inference budget), 0.79 (R ≈ 1), and 22 (R ≫ 1, many inference tokens — the budget is tight).

Revisions (Figure 9, left; Figure 1, top-right bar chart). The paper reports relative performance of the smaller model with compute-optimal revisions versus the ~14× larger model, broken out by difficulty grouping (aggregating bins for readability):

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy (bin 1)	+11.8%	+3.5%	−11.9%
Medium (bins 2–3)	+27.8%	+16.7%	+5.4%
Hard (bins 4–5)	+21.6%	(implied negative)	−37.2%

At R ≪ 1, test-time compute outperforms the larger model across all difficulty levels. At R ≫ 1, it only remains preferable on easy questions. The transition from advantage to disadvantage as R increases is monotonic and sharp — test-time compute is most valuable when inference volume is low relative to pretraining volume, consistent with the FLOP accounting where pretraining savings dominate when R is small.

PRM search (Figure 9, right; Figure 1, bottom-right bar chart). The pattern is starker than revisions:

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy	+19.1%	+2.2%	+2.0%
Medium	0.0%	−35.3%	−30.8%
Hard	−3.6%	−35.3%	−52.9%

PRM search shows substantially weaker benefits than revisions for the FLOPs-matched comparison, with large disadvantages on medium and hard questions even at moderate R values. On easy questions, test-time compute with PRM search remains preferable across all R regimes, though the margin narrows significantly from +19.1% (R ≪ 1) to +2.0% (R ≫ 1).

Figure 9 line plots. The accuracy-per-difficulty-bin curves as test-time compute scales are overlaid with stars representing the greedy performance of the ~14× larger model at the three R values. Where the compute-optimal scaling line is above the star, test-time compute wins. For revisions on Bin 1 (easiest, purple topmost line), the scaling line is above all three stars — test-time compute dominates across all R. For revisions on Bin 5 (hardest, blue bottommost line), the line is below all three stars and essentially flat near 0–5%, confirming that no amount of test-time compute helps on the hardest problems regardless of R.

Ablation Studies and Robustness Checks

PRM aggregation strategy (Appendix E, Figure 13). Comparing "min," "prod," and "last" step-wise score aggregation at 256 samples: "last" achieves roughly 37%, "min" achieves roughly 35%, "prod" achieves roughly 27%, and ORM achieves roughly 34%. The paper notes this contradicts prior work (Lightman et al., 2023; Wang et al., 2023) which found "min" to be best, hypothesizing that the discrepancy arises because their PRM is trained with soft Monte Carlo labels rather than binary correctness labels. The "last" aggregation effectively reduces the PRM to ORM-like behavior at selection time, yet the PRM still outperforms a separately trained ORM — evidence that step-level PRM training provides beneficial representation learning even when intermediate predictions aren't directly used at aggregation time.

PRM vs. ORM scaling (Appendix F, Figure 14). The PRM consistently outperforms the ORM, with the gap widening at higher sample counts. At 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35%. Majority voting trails both substantially (roughly 30%). This confirms that the PRM's training provides genuine signal improvement beyond what is achievable with final-answer-only supervision, and that this improvement scales — more samples widen the gap between PRM and ORM rather than saturating.

Revision model verifier choice (Appendix J, Figure 15a). The base-LM PRM — trained on PaLM 2-S* base model outputs — underperforms the revision-specific ORM when scoring revision model outputs: sequential + base-LM PRM achieves roughly 40% at 64 generations vs. sequential + revision ORM at roughly 42%. This confirms distribution shift as a practical concern: a verifier trained on one proposal distribution does not necessarily transfer to a different proposal distribution (the revision model's outputs differ systematically from the base model's).

Revision history in verifier context (Appendix J, Figure 15b). Including previous revisions in the ORM's context provides a small improvement over the no-history ablation (approximately 1–2 percentage points at 64 generations). Critically, both variants outperform the parallel baseline, confirming that the sequential sampling benefit is not solely attributable to the verifier seeing more context — the revision model genuinely produces better candidates through iterative refinement, not just richer information for the verifier to evaluate.

Oracle vs. predicted difficulty bins (Figures 4 and 8; Appendix C, Figures 11–12). Both oracle and predicted bins yield qualitatively similar trends across difficulty levels. In the search setting (Figure 4), the predicted and oracle curves "largely overlap," with predicted bins achieving essentially identical performance. In the revision setting (Figure 8), predicted bins show slightly lower performance at high budgets (approximately 41% vs. 44% at 256 generations) but still substantially outperform the parallel baseline. This is the critical robustness check enabling practical deployment: the compute-optimal strategy works without ground-truth labels.

Majority voting for revisions (Appendix B, Figure 10). The sequential-to-parallel ratio trends observed with verifier-based selection are replicated when using simple majority voting instead: easy questions are insensitive to ratio, hard questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate. The pattern is robust to the selection mechanism, demonstrating that the improvement from sequential revisions is in the quality of generated candidates, not merely in providing more context for a learned verifier.

ReST^(EM) revision model (Appendix K, Figure 16). An attempt to further optimize the revision model using ReST^(EM) (Singh et al., 2024) — which involves on-policy data collection and RL-style fine-tuning — backfires substantially. At 256 generations, fully sequential performance with the ReST^(EM) revision model drops to approximately 33.5% compared to roughly 38.5% at the optimal sequential-to-parallel ratio. The authors hypothesize that on-policy data collection exacerbates spurious correlations in revision data, causing the model to learn the wrong patterns. This is a notable negative result that highlights the sensitivity of revision training to the data generation procedure and cautions against straightforward application of RL fine-tuning to revision models.

Critical Assessment

Does the paper demonstrate that AI use impairs skill formation by 17% (Cohen's d=0.738)?

The experiment directly supports this claim with a properly powered, randomized between-subjects design (n=52, p=0.010, Figure 6). The effect size is substantial and significant. Several design features strengthen internal validity: balanced assignment across covariates (Table 1), screen-recording-based compliance verification, a psychometrically validated quiz instrument tested across five versions and four pilot studies (Section 5.1), and pre-registration of the grading rubric and disqualification criteria (Section 4.3). The exclusion of one participant was procedural (blank questions due to interface confusion) and pre-registered, reducing concerns about selective exclusion.

However, there are important limits to the claim's generality. First, the task is a single, specific library (Trio) learned over approximately one hour — the paper measures immediate skill formation, not long-term retention or transfer. The authors acknowledge this explicitly (Section 7.1), noting that "skill formation takes place over months to years." Whether the impairment persists, attenuates, or compounds over longer timescales is unknown. Second, the AI assistant is a chat-based interface — the authors argue this is a "lower bound for cognitive offloading since agentic AI coding tools would require even less human participation" (Section 7.1), but this is a hypothesis rather than a demonstrated fact. Third, the quiz measures declarative knowledge (conceptual understanding, code reading, debugging), not procedural fluency — the paper cannot speak to whether AI-impaired participants would perform differently on subsequent coding tasks requiring Trio. Fourth, the flat-rate compensation structure may not generalize to workplace settings where productivity incentives and organizational pressures shape behavior differently.

Does the paper demonstrate that AI use did not significantly improve task completion time?

Yes, but with important heterogeneity that the aggregate null result masks. The overall difference in completion time is non-significant (Figure 6), but the qualitative analysis reveals why: only a subset of AI users (those who directly pasted generated code, n=9) completed tasks substantially faster than the control group (Figure 13). Other AI users spent their time savings on composing queries, reading explanations, or manually typing AI-generated code — these behaviors preserved or consumed time. The finding is genuine but conditional: AI can accelerate task completion, but only for specific usage patterns, and the usage patterns that maximize speed (direct delegation, pasting) are also associated with the worst learning outcomes. The authors do not test whether time-equated conditions would show different learning effects — e.g., giving the AI group less total time to match the control group's effective coding time.

Does the paper demonstrate that certain AI interaction patterns preserve skill formation?

This is supported by the qualitative analysis but with important methodological caveats. The six personas (Figure 11) are derived from post-hoc clustering of observed behaviors, not from pre-registered hypotheses or random assignment to interaction patterns. The sample sizes within each persona are small (n=2 to n=7), and no statistical tests are reported comparing persona-level outcomes. The analysis is descriptive and hypothesis-generating, not causally identified — we cannot conclude from this data alone that adopting a high-scoring interaction pattern would cause better learning outcomes, because the participants who chose those patterns may differ systematically from those who chose delegation patterns in ways not captured by the pre-screening covariates (e.g., intrinsic motivation, learning orientation, metacognitive strategy).

That said, the pattern is striking and consistent with the theoretical framework. The 40+ percentage point gap between high- and low-scoring patterns (24–39% vs. 65–86%) within the AI-using group is larger than the overall treatment-control difference (17%), suggesting that how AI is used matters more than whether it is used. The paper appropriately treats these findings as exploratory and does not claim causal identification of persona effects.

Does the paper demonstrate that encountering errors is the mechanism by which AI impairs skill formation?

The paper provides strong correlational evidence but not direct causal evidence for this mechanism. The chain of observations is compelling: the AI group encountered fewer errors (Table 4), especially fewer Trio-specific errors (Figure 15), and the largest quiz score gap between conditions was on debugging questions (Figure 8). The theoretical link — that encountering and independently resolving Trio-specific errors builds conceptual understanding — is well-motivated by experiential learning theory.

However, the paper does not experimentally manipulate error frequency or error resolution mode to test this mechanism directly. An experiment that, for example, randomly assigned AI-condition participants to receive either immediate fixes or delayed hints when encountering errors would provide stronger causal evidence. The current design cannot rule out alternative mechanisms: perhaps AI-impaired learning occurs because participants spend less time reading Trio documentation (not because they encounter fewer errors), or because the AI's correct code provides a "completeness illusion" that reduces motivation to understand details. The error-frequency and error-type data are consistent with the mechanism but do not isolate it.

Does the paper provide actionable guidance for practitioners?

Yes, but with scope limitations that should be stated clearly. The persona taxonomy provides specific, identifiable patterns that are associated with learning preservation: ask only conceptual questions (Conceptual Inquiry), request explanations alongside code (Hybrid Code-Explanation), or generate first then interrogate (Generation-Then-Comprehension). The paper also identifies patterns to avoid: fully delegating coding tasks (AI Delegation), progressively surrendering to AI over the task (Progressive AI Reliance), or using AI as a debugging oracle without understanding fixes (Iterative AI Debugging).

However, these patterns were observed in a specific context — a chat-based interface, a one-hour learning task, a library explicitly designed for ease of learning, professional developers with prior AI experience, and flat-rate compensation. Whether the same patterns would emerge (or be effective) with agentic coding tools, over longer learning periods, with different types of tasks (e.g., debugging unfamiliar codebases rather than learning new libraries), or under different incentive structures is unknown. The paper appropriately frames these as directions for future work (Section 7.1) rather than established prescriptions.

What is missing?

Several experiments would have strengthened the paper's claims but were not run:

A longitudinal follow-up measuring skill retention days or weeks after the experiment would address whether the learning impairment persists or whether AI users eventually catch up through subsequent exposure. This is the most important missing experiment for the paper's practical implications — if the impairment is temporary, the case for restricting AI during learning is weaker.
A transfer task requiring participants to apply Trio knowledge to a novel problem (beyond the quiz) would test whether the learning differences are robust or merely reflect quiz-specific memorization. The current evaluation measures recall and recognition in formats similar to the task experience; genuine skill transfer is not tested.
A second model family to test whether the quantitative results generalize beyond GPT-4o. The paper's AI assistant is a single model with a single prompt; different models or prompting strategies might produce different usage patterns (e.g., a model that defaults to providing explanations alongside code might push more users toward Hybrid Code-Explanation).
A within-subjects component where participants experience both AI and no-AI conditions on different tasks would enable estimating individual-level treatment effects and testing whether learning impairment varies with participant characteristics.
An explicit manipulation of the AI assistant's behavior — e.g., one condition where the assistant provides only explanations and refuses to generate code, another where it provides code without explanation — would directly test whether the interaction patterns are causally responsible for learning differences, rather than merely correlated with them.

Summary of evidential support. The paper's central claim — that AI assistance during a learning task impairs skill formation on that task — is well-supported by a carefully designed, properly powered randomized experiment with strong compliance verification and psychometrically validated measurement. The paper's more ambitious claim — that specific interaction patterns preserve learning and that errors are the mechanism — is supported by qualitative evidence and theoretical reasoning but falls short of causal demonstration. The practical guidance is actionable but should be treated as empirically grounded hypotheses rather than established best practices. The paper is transparent about these limitations (Section 7.1) and appropriately frames itself as "a first step to understanding the impact of AI assistance on humans in the human-AI collaboration process."

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted For in the Headline 4× Efficiency Gain

The assumption or constraint. The entire compute-optimal framework rests on knowing each prompt's difficulty before deciding how to allocate the inference budget. The paper's method for estimating difficulty — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

Generating 2048 samples per question consumes more compute than the largest test-time budgets studied (256–512 generations). When the cost of difficulty estimation exceeds the cost of the strategy it selects, the claimed efficiency gains are measured from the wrong baseline.

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. For example, if difficulty estimation costs 2048 generations and the compute-optimal strategy uses 64 generations, the total cost is ~2100 generations — roughly the same as simply running best-of-N with 2048. The 4× figure should be understood as an upper bound on achievable efficiency conditional on knowing difficulty, not a realized deployment gain.

What evidence exists in the paper. The paper does not measure total cost including difficulty estimation. Figure 4 and Figure 8 both report performance as a function of the execution budget only, with the x-axis showing generations from 1 to 512. The difficulty estimation cost of up to 2048 generations per question is entirely outside the frame of these plots. The authors flag this in Section 3.2 but do not provide even a back-of-the-envelope calculation of what the inclusive cost would be or how the efficiency comparison would change.

Mitigation status. The paper discusses this as a key avenue for future work (Section 8), suggesting "pretraining or finetuning models to directly predict difficulty of a question" as the primary solution. It also mentions adaptive difficulty estimation as an alternative — starting with a small number of samples and adjusting strategy dynamically — but no experiments test this approach. Until cheap difficulty estimation is demonstrated, the compute-optimal framework is a theoretical contribution with a practical deployment gap: it shows what the optimal allocation looks like but not how to achieve it affordably. Practitioners wanting to deploy compute-optimal scaling would need to either (a) develop their own cheap difficulty estimator, (b) accept the difficulty estimation cost as a one-time pre-computation amortized over many similar queries (which the paper does not evaluate), or (c) accept that the 4× gain is smaller (potentially much smaller) when difficulty estimation cost is included.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The choice of MATH is deliberate — mathematical reasoning requires multi-step logical deduction rather than factual recall, making it ideal for studying test-time compute — but the paper provides no evidence that the findings generalize. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not a finding.

The consequence. Multiple aspects of the findings could be model-specific or task-specific:

PRM over-optimization behavior (Figure 3, right — beam search degrading on easy problems at high budgets) depends on the PRM's calibration properties, which are a function of the base model's output distribution. A model with different error patterns (e.g., systematically different types of mistakes on easy problems) could exhibit different over-optimization thresholds, changing which difficulty bins benefit from search versus best-of-N.
The difficulty-dependent optimal sequential-to-parallel ratio for revisions (Figure 7, right) depends on the revision model's ability to produce improved candidates across multiple steps, which varies with the base model's architecture, training data, and scale.
The FLOPs-matched comparison (Section 7) depends critically on the ~14× larger model's performance, which is a function of PaLM 2's specific scaling properties. A model family with different scaling exponents could yield different crossover points between test-time and pretraining compute.
MATH problems have unambiguous correct answers and can be graded with exact string matching. Tasks without clean correctness signals (code generation evaluated via unit tests, open-ended generation, dialogue) would require fundamentally different verifier training and difficulty estimation approaches that the paper does not develop.

What evidence exists in the paper. The paper provides no cross-model or cross-benchmark experiments. The test set of 500 questions is split into quintiles of ~100 each, further split by two-fold cross-validation, meaning the compute-optimal policy is selected based on ~50 questions per fold per bin — a small sample. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the selected strategies would remain optimal on a different 500-question sample from the same distribution, let alone a different benchmark.

Mitigation status. The paper does not attempt to address this limitation empirically. Section 8 mentions extending the analysis to other domains (code generation, logical reasoning) as future work but provides no preliminary evidence. Practitioners applying these findings to different models, benchmarks, or domains should expect that the qualitative patterns (difficulty-dependence, over-optimization at high budgets, complementary strengths of search and revisions) may transfer, but the quantitative thresholds (which difficulty bin benefits from which strategy, at what budget level) almost certainly will not.

The FLOPs-Matched Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining-Vs-Inference Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal paradigm (Hoffmann et al., 2022) where both data and parameters are scaled equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the ~14× larger model is evaluated with only greedy decoding — no majority voting, no best-of-N, no test-time compute of any kind. This makes the comparison asymmetric: the smaller model is given the benefit of compute-optimal test-time allocation, while the larger model gets no inference-time augmentation whatsoever.

The consequence. Both design choices make the pretraining baseline weaker than it needs to be, potentially overstating the advantage of test-time compute:

A Chinchilla-optimal model — one that allocates extra pretraining FLOPs to both more parameters and more training data — would likely outperform a parameter-only-scaled model at the same total pretraining budget. The paper's reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R ≪ 1 for revisions, shown in Figure 9 and Figure 1) may shrink or reverse against a properly compute-optimal larger model.
Even a modest amount of test-time compute applied to the larger model (e.g., best-of-8) could close or reverse the gap on medium-difficulty questions, where the paper currently reports test-time compute as competitive or better at moderate R values. If the larger model also benefits from compute-optimal scaling, the crossover point where test-time compute becomes preferable would shift toward lower R values or easier difficulty bins.

What evidence exists in the paper. The paper reports the ~14× larger model's performance using only greedy decoding (stars in Figure 9). There is no experiment giving the larger model any test-time compute budget, no comparison against a Chinchilla-optimal baseline, and no analysis of how the results would change if the larger model received even a modest inference budget. The paper includes this caveat in Section 7 but treats the parameter-only scaling as an acceptable baseline for a "first analysis" rather than investigating how the comparison would change with stronger baselines.

Mitigation status. The paper explicitly acknowledges the limitation in Section 7 and frames the Chinchilla-optimal comparison as future work. This is a fair scoping decision for an initial study, but a practitioner deciding how to allocate compute between training and inference should treat the paper's FLOPs-matched results as a lower bound on pretraining's advantage — the true advantage of scaling pretraining compute is almost certainly larger than what Figure 9 reports, because the paper's pretraining baseline understates what a well-allocated pretraining budget can achieve.

Verifier Over-Optimization Remains Unsolved and Sets a Hard Ceiling on Test-Time Compute Scaling

The assumption or constraint. The paper identifies PRM over-optimization as the primary bottleneck preventing unbounded improvements from additional test-time compute — beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search paradoxically performs worst overall (Figure 3, left), and qualitative examples show search producing degenerate outputs (repetitive steps, overly short solutions; Appendix M). The compute-optimal policy mitigates this by routing easy problems away from aggressive search (using best-of-N instead of beam search for Bins 1–2), but it does not solve the underlying problem.

The consequence. On medium-difficulty problems where beam search is deployed (Bin 3–4), over-optimization still limits the scaling ceiling. The beam search curves in Figure 3 (right) for Bins 3–4 flatten and, in some cases, begin declining well before the maximum budget of 256 generations is exhausted. This means that scaling test-time compute beyond a certain point yields zero or negative returns, and the compute-optimal policy cannot circumvent this — it can only choose the best available strategy at each budget, and when all strategies are plateauing or degrading, there is no good option. The paper's findings on test-time compute are therefore specific to the verifier quality achievable with the Monte Carlo rollout training procedure described in Appendix D. A verifier with better calibration, adversarial robustness, or ensemble-based scoring could shift the over-optimization threshold outward, potentially changing both the optimal strategy per bin and the maximum useful budget.

What evidence exists in the paper. Figure 3 (right) shows beam search degrading or plateauing on Bins 1–2 (over-optimization) and plateauing on Bins 3–4 (verifier signal saturation). Figure 3 (left) shows lookahead search — the most powerful optimizer — underperforming all other methods, providing a clean demonstration that stronger optimization of an imperfect verifier reduces performance. Appendix M (Figure 29, etc.) provides qualitative examples of degenerate search outputs. The paper does not experiment with any verifier improvements (better training data, ensemble methods, calibration techniques) to test how the scaling landscape would change.

Mitigation status. The paper treats this as a research direction rather than an attempted solution. Section 8 discusses "improving verifier robustness" as a key future direction, suggesting adversarial training (training the PRM on search-generated rather than i.i.d. solutions), ensemble verification, or KL-penalty approaches analogous to RLHF. The paper's contribution is to document the over-optimization phenomenon clearly and demonstrate that it is the binding constraint, but it offers no path to overcoming it. Practitioners deploying test-time compute at scale should expect that the useful budget is bounded by verifier quality, and that improving the verifier — not developing more sophisticated search algorithms — is the highest-leverage investment for pushing the scaling frontier further.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate and Revision Training Is Fragile

The assumption or constraint. The revision model, fine-tuned on trajectories where all in-context answers are incorrect followed by a correct target, has no training signal for what to do when the current answer is already correct. At test time, the model encounters correct answers in its own revision chain (produced during earlier steps) and sometimes incorrectly "revises" them into wrong answers. Section 6.1 reports that approximately 38% of correct answers get converted back to incorrect ones using a naive approach.

The consequence. This reversion problem means that longer revision chains have a self-undermining property: the model improves answers over several steps, but past a certain point, the probability of incorrectly reverting a correct answer counteracts the probability of improving an incorrect one. The paper mitigates this with majority voting or verifier-based selection across the entire chain (picking the best answer from any point), but these are imperfect patches — they require running the full chain and then discarding later revisions that are worse than earlier ones, wasting compute on unproductive steps. A more principled solution — such as training the model to recognize when no revision is needed, or to output a "stop" token — is not explored.

The fragility of revision training is further demonstrated by the ReST^(EM) experiment (Appendix K, Figure 16). Attempting to optimize the revision model with RL-style on-policy training caused performance to degrade substantially: at 256 generations, fully sequential performance with the ReST^(EM) revision model drops to approximately 33.5% compared to roughly 38.5% at the optimal sequential-to-parallel ratio with the SFT revision model. The authors hypothesize that on-policy data collection amplifies spurious correlations in revision trajectories, but the precise mechanism is not diagnosed.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1 without a supporting figure or table — it appears as a textual claim. The ReST^(EM) degradation is shown in Figure 16 (Appendix K). The paper does not provide a per-step analysis of reversion probability (how it changes over the revision chain), does not compare reversion rates across difficulty bins, and does not test alternative mitigation strategies beyond the within-chain selection mechanisms already used.

Mitigation status. The paper treats within-chain selection (majority voting or verifier-based selection) as a sufficient mitigation, but this is a workaround, not a solution — it discards computation spent on reverted steps without addressing the root cause. The ReST^(EM) failure is noted as a cautionary finding but not deeply investigated. A practitioner deploying revision models would need to either (a) implement within-chain selection at the cost of wasted compute on reverted steps, (b) develop a stopping criterion (which the paper does not provide), or (c) accept that revision chains have a useful length limit beyond which additional revisions are net harmful. The paper's reported sequential revision benefits (Figure 6, Figure 7) are likely specific to the SFT training procedure with edit-distance-based data construction and may not transfer to other training recipes — the ReST^(EM) result demonstrates that superficially similar training approaches can produce qualitatively different revision behavior.

The Hardest Problems Are Essentially Unsolved — Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. Across all methods — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty Bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), Bin 5 accuracy hovers at 1–3% for all search methods at all budgets. In Figure 7 (right), Bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the Bin 5 scaling line is essentially flat near 0–5% for both revisions and search. The paper is transparent about this:

"On the hardest questions (bin 5), no method makes meaningful progress. No allocation strategy helps."

The consequence. Test-time compute amplifies existing capability but does not create it from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path to improvement.

This limitation is particularly important for evaluating the pretraining-vs-inference tradeoff. The paper's FLOPs-matched comparison (Figure 9) shows that test-time compute with the smaller model outperforms the ~14× larger model on easy questions across all R values, but underperforms on hard questions — sometimes dramatically (e.g., −52.9% relative disadvantage for PRM search on hard questions at R ≫ 1). Organizations dealing with problem distributions that include substantial fractions of genuinely hard problems (where the base model's pass@1 is near zero) should expect test-time compute to provide minimal benefit and should prioritize pretraining investment instead.

What evidence exists in the paper. The Bin 5 results are consistently near-zero across all figures: Figure 3 (right) for search, Figure 7 (right) for revisions, Figure 9 for the FLOPs-matched comparison. The paper explicitly states this as a boundary condition in the Section 7 takeaway box. However, the paper does not characterize what makes Bin 5 problems hard — whether they require knowledge not present in the base model's training data, whether they involve reasoning chains too long for the model to produce even with search, or whether they require conceptual leaps the model cannot make. Understanding the nature of the capability gap would help practitioners predict whether their domain's hard problems are likely to be in the "unsolvable by test-time compute" regime.

Mitigation status. The paper treats this as an inherent limitation of test-time compute rather than a solvable problem within the current framework. The authors do not propose methods for extending test-time compute to Bin 5 problems, suggesting instead that such problems require pretraining investment. This is a reasonable scoping decision — the paper's contribution is to characterize where test-time compute helps and where it doesn't, and the Bin 5 finding provides a clear boundary condition. Practitioners should use this as a diagnostic: if current pass@1 on the target problem distribution is near zero, invest in better pretraining rather than smarter inference, because no test-time strategy will create correct solutions that the base model cannot produce at any non-trivial rate.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes the conversation around AI in the workplace from a product-centric to a process-centric evaluation. Prior to this work, the dominant empirical paradigm for studying AI assistance measured what workers produced — lines of code, tasks completed, pull requests merged — and found substantial gains, particularly for novices (Peng et al., 2023; Cui et al., 2024; Brynjolfsson et al., 2025). This paper does not dispute those findings but demonstrates that they are diagnostically incomplete: measuring only output obscures what happens to the worker during AI-assisted task completion. The paper's core empirical contribution — a 17% reduction in skill formation (Cohen's d=0.738) with no significant productivity gain — establishes that the process of using AI during learning carries costs invisible to product-only evaluation.

This is not merely a "productivity versus learning" tradeoff finding. It is a diagnostic about the wrongness of the prevailing evaluation target. The paper shows that productivity and skill formation are not always aligned, and that optimizing for the former — as organizations with short-term metrics do — can degrade the latter without triggering any alarm in productivity dashboards. The persona analysis (Figure 11) sharpens this point: the fastest interaction pattern (AI Delegation, averaging 19.5 minutes) was also among the lowest-scoring on the quiz (24–39%), while the fastest high-learning pattern (Conceptual Inquiry) was the second-fastest overall. This means the relationship between speed and learning is not a simple monotonic tradeoff — some ways of using AI are both fast and learning-preserving, while others are fast but learning-destructive. The diagnostic contribution is to show that the specific interaction pattern, not AI access per se, mediates the outcome.

The paper also reconciles contradictory findings across adjacent literatures. Productivity studies (Peng et al., 2023; Cui et al., 2024) found that AI accelerates coding; this paper finds no significant acceleration overall. The apparent contradiction is resolved by the qualitative analysis (Section 6.1): only a subset of users — those who directly pasted AI-generated code — experienced substantial speed gains. Other users invested their time savings in query composition, explanation reading, and manual code copying, resulting in completion times comparable to the control group. The productivity literature's findings are not wrong but are averaged over a heterogeneous set of user behaviors that the persona taxonomy disaggregates. Similarly, cognitive offloading surveys (Gerlich, 2025; Lee et al., 2025) found associations between AI use and reduced critical thinking, but could not establish causality — the randomized design converts this correlation into a causal claim, showing that AI use causes measurable skill impairment, not merely correlates with it.

The most unsettling implication is the exoskeleton reversal that the paper documents. Wiles et al. (2024) showed that AI acts as an exoskeleton — providing capabilities that disappear when the tool is removed. This paper shows the exoskeleton not only fails to transfer but proactively atrophies the muscles it was meant to support. This is distinct from skill retention (where learned skills don't survive AI removal) — it is skill acquisition prevention, where skills never form in the first place because the learning process is short-circuited. This reframes the concern: it is not that AI creates dependency but that AI prevents the development of the expertise needed to escape dependency.

The paper also introduces a new mechanism for AI-induced skill impairment — error encounter reduction — that connects observable behavior to learning outcomes. The finding that the AI group encountered dramatically fewer Trio-specific errors (Figure 15) and subsequently performed worst on debugging questions (Figure 8) operationalizes experiential learning theory in an empirically testable way. This is not just a pedagogical observation; it is a causal hypothesis about how AI tools disrupt skill formation: by eliminating the concrete experiences (errors, diagnostic reasoning, independent resolution) that drive conceptual understanding. This mechanism is specific, testable, and suggests concrete interventions — tools that help learners engage with errors rather than eliminating them.

This finding redirects research attention in several ways. It makes interaction design for learning a first-order concern rather than an afterthought. It shifts the question from "does AI help or hurt?" to "which AI interaction patterns preserve learning, and how can tools be designed to encourage them?" It elevates qualitative process analysis — screen recording annotation, interaction pattern clustering — from supplementary illustration to a necessary empirical method for understanding AI's developmental effects. And it makes the speed-versus-learning tension empirically visible and measurable, providing the foundation for organizations to design AI deployment policies that balance short-term productivity with long-term workforce capability.

The paper does not argue for restricting AI use in learning contexts. It argues for intentional design of AI-assisted learning — a fundamentally different posture than either unrestricted adoption or blanket prohibition. This is the paper's most durable conceptual contribution: a framework in which the question is not whether to use AI for learning, but how, with specific, empirically identified patterns that work and don't work.

Follow-Up Research This Work Enables

1. Causal testing of interaction patterns through experimental manipulation of AI assistant behavior. The six personas (Figure 11) are post-hoc clusters derived from observational analysis of screen recordings — they describe correlations between spontaneously adopted behaviors and learning outcomes, but do not establish that adopting a high-scoring pattern causes better learning. A strong follow-up would randomly assign participants to AI assistants with different behavioral constraints: (a) a "delegation" condition where the assistant provides only complete code solutions without explanation, (b) a "conceptual inquiry only" condition where the assistant refuses to generate code and only answers conceptual questions, (c) a "generation-then-comprehension" condition where the assistant provides code but then prompts the user with a follow-up comprehension question before allowing further interaction, and (d) an unrestricted baseline. The dependent variables would be the same quiz instrument from this paper, plus task completion time. This design would directly test whether the interaction patterns themselves — rather than unobserved user characteristics — cause the learning differences. The paper's quiz and task materials are publicly available, making this design straightforward to implement.

2. Longitudinal measurement of skill retention and transfer beyond the immediate post-task quiz. The paper measures skill formation through a quiz administered minutes after task completion (Section 4.2). This captures immediate learning but cannot speak to whether the AI-induced impairment persists, attenuates, or compounds over time. A longitudinal study would bring participants back after 1 day, 1 week, and 1 month for follow-up assessments: (a) the same quiz to measure retention, (b) a novel Trio task requiring application of the learned concepts to a new problem (measuring transfer), and (c) a task requiring participants to debug and modify AI-generated Trio code (directly testing the supervision capability the paper identifies as critical). If the gap closes over time — perhaps because control group participants forget while AI group participants catch up through subsequent experience — the practical implications would be weaker. If the gap persists or widens — because the AI group never acquired the foundational concepts — the case for restricting AI during initial learning would be much stronger. The paper's qualitative finding that control group participants reported higher self-reported learning (Figure 9) and more positive feedback (Tables 6–7) hints that experiential differences may compound motivationally, not just cognitively.

3. Comparison with human assistance as a counterfactual learning intervention. The paper explicitly identifies the absence of a human-assistance condition as a limitation (Section 7.1):

"We do not include the counterfactual of how skill formation would be impacted by receiving assistance from humans."

This is not merely a missing control condition — it is a theoretically crucial comparison because different forms of assistance may have different effects on skill formation. A human tutor, unlike an AI assistant, can calibrate help to the learner's zone of proximal development, withhold answers when the learner can solve independently, and ask diagnostic questions that promote understanding rather than delegation. A study with three conditions — no assistance, AI assistance (replicating this paper's treatment), and human assistance (e.g., a senior developer available via chat) — would reveal whether the skill impairment is specific to AI assistance or inherent to any form of help during learning. If human assistance produces similar impairment, the problem is assistance per se, and the solution is to delay all assistance during initial skill acquisition. If human assistance preserves or enhances learning, the problem is specific to how current AI tools provide assistance — they solve problems for users rather than helping users learn to solve problems — and the solution is to redesign AI tools to mimic effective human tutoring behaviors (scaffolding, Socratic questioning, progressive hint disclosure).

4. Agentic coding tools and the cognitive engagement floor. The paper's AI assistant is a chat-based interface requiring users to compose queries, read responses, and decide what to do with AI output. The authors argue this is a "lower bound for cognitive offloading since agentic AI coding tools would require even less human participation" (Section 7.1). This is a testable hypothesis. An experiment identical in structure to this paper, but comparing a chat-based assistant (replication) with an agentic coding tool (e.g., Cursor's agent mode, GitHub Copilot's agent features) would measure whether the skill impairment scales with automation level. The agentic condition would likely show: (a) even fewer errors encountered (the agent catches and fixes errors autonomously before the user sees them), (b) even less active coding time (Figure 16 — the agent writes code while the user watches), (c) possibly faster task completion (reducing query composition overhead), and (d) correspondingly larger skill impairment. The paper predicts this directionally but provides no data. A negative result — agentic tools producing similar or smaller impairment — would be surprising and important, suggesting that increased automation somehow triggers compensatory learning strategies (e.g., users monitoring more carefully because they trust the agent less).

5. Organizational incentive structures and the speed-versus-learning dynamic. The paper's participants were paid a flat rate and instructed to "complete the task as fast as they could" (Section 4.3) — a specific incentive structure that may not generalize to workplace settings where performance reviews, promotion prospects, and team norms shape AI usage. A field experiment within a software organization — randomizing junior developers to receive different AI usage guidelines (e.g., "use AI however you want to maximize PR throughput" vs. "use AI for explanations only; write all production code yourself" vs. no explicit guidance) and measuring both short-term productivity metrics (PR volume, cycle time) and medium-term skill development (code review quality, independent debugging capability, promotion velocity) — would test whether organizational policies can shift the distribution of interaction patterns toward skill-preserving modes. The paper's persona taxonomy provides a measurement framework for such a study: screen recordings could be annotated using the same event taxonomy (Table 5), and developers' interaction patterns could be correlated with downstream outcomes. The key question is whether the fastest interaction patterns (AI Delegation, direct pasting) are so attractive under production pressure that voluntary guidelines are insufficient to shift behavior — if so, tool-level constraints (e.g., requiring explanation queries before code generation) may be necessary.

6. Domain generalization and the specificity of error-driven learning. The paper finds that the largest skill impairment occurs on debugging questions (Figure 8) and that the mechanism involves reduced Trio-specific error encounters (Figure 15). This mechanism depends on the learning domain having diagnostic errors — errors whose resolution requires understanding domain concepts, not just surface-level fixes. Not all domains have this property. For tasks where errors are primarily syntactic (e.g., learning a new natural language grammar) or where correctness is subjective (e.g., learning design principles), the error-encounter mechanism may not operate, and AI assistance may have different (possibly smaller) effects on skill formation. A replication across diverse learning domains — code learning with a conceptually novel library (conceptual errors → large impairment, replicating this paper), code learning with a syntactically novel but conceptually familiar library (fewer conceptual errors → smaller impairment), data analysis with a new visualization library (design errors are subjective → impairment pattern differs), and writing in a new genre (no "errors" in the traditional sense → possibly no impairment or different mechanisms) — would map the boundary conditions of the paper's findings and refine the theoretical mechanism.

Practical Applications and Downstream Use Cases

1. Onboarding workflows for junior software developers. The paper's most directly actionable finding for engineering organizations is that certain AI interaction patterns preserve learning while others destroy it. An onboarding program for junior developers learning a new internal library or codebase could mandate specific AI usage modes: (a) require that all AI queries include an explicit request for explanation alongside any generated code (enforcing the Hybrid Code-Explanation pattern identified in Figure 11 as achieving ~65–86% quiz scores), (b) require that developers attempt to resolve errors independently for at least 5–10 minutes before querying AI for debugging help (preserving the error-encounter mechanism identified in Section 6.2), and (c) prohibit direct pasting of AI-generated code during the first two weeks (enforcing manual code copying, which Figure 13 shows does not improve learning but prevents the fastest, most learning-destructive Delegation pattern). The paper's finding that the median control group participant encountered 3 errors versus 1 in the AI group (Table 4) provides a concrete metric: if onboarding workflows produce error encounter rates closer to the control group's distribution, they are likely preserving skill formation. Organizations adopting these guidelines would need to monitor whether they reduce short-term task throughput — the paper's null productivity result suggests they may not, since manual copying and query composition consumed the time savings from AI code generation.

2. AI coding tool configuration for educational and training contexts. The paper's persona taxonomy directly informs tool design. Current AI coding assistants (GitHub Copilot, Cursor, Codeium) default to providing complete code completions with minimal explanation, which maps most closely to the Delegation pattern associated with the worst learning outcomes (24–39% quiz score). An educational mode for these tools — activatable by instructors or self-selected by learners — could implement constraints derived from the high-scoring patterns: (a) refuse to generate complete function implementations, instead providing only conceptual guidance and syntax hints (emulating Conceptual Inquiry, the fastest high-scoring pattern), (b) when generating code, always include inline explanatory comments and require the user to answer a comprehension question before the code is inserted (emulating Generation-Then-Comprehension), or (c) when the user pastes an error, provide a diagnostic question rather than a fix (e.g., "What does it mean when Trio raises a RuntimeWarning? What could cause this in your code?"), forcing the diagnostic reasoning that the control group experienced through independent error resolution. The paper's finding that debugging questions showed the largest score gap between conditions (Figure 8) suggests that tools' default behavior of fixing errors immediately is particularly damaging to skill formation and should be the first target for redesign.

3. Hiring and skill assessment in the AI era. The paper's finding that AI use during learning impairs subsequent unaided performance on debugging and conceptual understanding questions has implications for technical interviewing. If candidates have used AI extensively during learning — which the paper's recruitment data suggests most have (all participants had tried AI coding assistance "at least a few times," Section 4.3) — their unaided debugging and code reading abilities may be weaker than their experience level suggests, even if their AI-augmented productivity is high. Organizations hiring for roles requiring independent code review, debugging of AI-generated systems, or safety-critical software verification should consider: (a) including questions that specifically test conceptual understanding and debugging without AI assistance (similar to the paper's quiz), since these skills appear most vulnerable to AI-induced atrophy, (b) asking candidates about their AI usage patterns during learning — the persona taxonomy (Figure 11) provides a vocabulary for this conversation, and (c) recognizing that candidates who learned primarily through AI Delegation may have inflated productivity on simple tasks but substantial gaps in the foundational understanding needed for novel or safety-critical work. The paper's quiz instrument (14 questions, 27 points, covering debugging, code reading, and conceptual understanding) provides a template for domain-specific skill assessments that measure what AI-assisted learning may have left undeveloped.

4. Regulatory and professional certification frameworks. As AI coding tools become ubiquitous, professional bodies overseeing software engineering licensure or certification may need to update their competency standards. The paper's demonstration that AI assistance during learning impairs exactly the skills needed for AI supervision — debugging AI-written code, understanding whether generated code uses appropriate design patterns, verifying correctness — creates a regulatory concern: if certification exams allow AI assistance, they may certify developers who cannot perform the supervision tasks that their professional role requires. Conversely, if exams prohibit AI assistance but developers have trained primarily with AI, exam performance may decline even as workplace productivity increases — creating a divergence between certification standards and workplace realities. The paper's finding that the quiz (a no-AI assessment) revealed substantial gaps in the AI-trained group (17% score reduction) suggests this divergence is already measurable. Professional bodies could use instruments modeled on the paper's quiz to establish baseline competency requirements for AI-supervision skills and to evaluate whether current training pathways — which increasingly incorporate AI — are producing developers who meet those requirements.

When to Prefer AI-Assisted Learning with Cognitive Engagement Constraints vs. AI-Free Learning vs. Unrestricted AI Use

The paper does not propose a single method that competes against named alternatives. Instead, it identifies a spectrum of AI interaction patterns with sharply different learning outcomes. The decision rule that emerges from the paper's findings can be stated as a set of context-dependent recommendations grounded in the specific empirical results:

Prefer AI-free learning (control condition) when:

The learning target involves foundational skills that will be needed for supervising, debugging, or verifying AI outputs in the future (Section 6.2 — the largest skill gap was on debugging, exactly the skill needed for AI oversight).
The task involves concepts where the primary learning mechanism is encountering and independently resolving errors (Figure 15 — AI use nearly eliminated Trio-specific error encounters, which drove conceptual understanding in the control group).
The learner is in an early career stage and the skills being acquired are foundational to their professional development (Section 7 — the paper explicitly warns that junior workers relying on AI for speed may sacrifice long-term competence).
The organization or individual prioritizes long-term expertise development over short-term task throughput, and the specific task does not require knowledge the learner completely lacks (the paper's control group completed tasks at comparable speed to the AI group overall, Figure 6).

Prefer AI-assisted learning with cognitive engagement constraints when:

AI access cannot be fully restricted (e.g., the learner is remote, the organization uses AI tools broadly) but the learning goal is important.
The interaction can be constrained to one of the three high-scoring patterns: Conceptual Inquiry (only ask conceptual questions), Hybrid Code-Explanation (request explanations alongside generated code), or Generation-Then-Comprehension (generate code, then interrogate the AI about it). These patterns achieved 65–86% quiz scores in Figure 11, comparable to the control group.
The learner can be trained or nudged to avoid the three low-scoring patterns: AI Delegation (fully relying on AI to write code), Progressive AI Reliance (starting with questions then surrendering), and Iterative AI Debugging (repeatedly asking AI to fix errors without understanding). These patterns achieved 24–39% quiz scores.
The AI tool can be configured to refuse complete code generation, require comprehension checks before providing answers, or provide diagnostic questions rather than fixes when errors occur.

Prefer unrestricted AI use when:

The task exclusively requires existing skills that the worker already possesses — the paper's domain is skill formation, not skill application, and its findings do not speak to whether AI impairs performance on tasks using already-mastered knowledge.
The task is a one-time production need where speed is paramount, skill formation is irrelevant, and the output will be verified by someone who already has the relevant expertise (e.g., a senior developer reviewing AI-generated code from a tooling script that will not be maintained).
The alternative is task non-completion — the paper notes that 4 of 26 control participants did not finish the second task within the 35-minute limit, while all AI participants finished both tasks (Section 5.2.2). When the choice is between completing a task with AI and not completing it at all, AI use is preferable regardless of learning effects.

The paper does not directly test these decision rules — they are inferences from the experimental results applied to practical contexts. The persona sizes are small (n=2 to n=7), and the interaction patterns were spontaneously adopted rather than experimentally assigned, so the causal effect of imposing a specific pattern remains untested. Organizations adopting these recommendations should treat them as empirically grounded hypotheses to be validated in their specific context, not as established best practices with guaranteed outcomes.