Learning Classifiers on Positive and Unlabeled Data with Policy Gradient

ArXiv: 1910.06535

🎯 Pitch

Instead of fixing label guesses before training, this paper has a classifier and a label-guessing policy teach each other through reinforcement learning—producing a classifier from only positive and unlabeled data that nearly matches the performance of one trained with perfect negative labels.

1. Executive Summary

This paper proposes policyPU, a reinforcement learning framework that jointly trains a policy network and a binary classifier from positive and unlabeled (PU) data, replacing the conventional two-step pipeline—where class prior or label noise is estimated first and a classifier is built second—with an end-to-end interactive process. The policy network acts as an agent that infers label assignments for unlabeled examples (producing either continuous soft labels as a Weighter or discrete hard assignments as a Separator), while the classifier provides classification coherence rewards that guide policy improvement via REINFORCE, enabling the two components to dynamically adapt to each other throughout training. Evaluated on MNIST, CIFAR-10, and an e-commerce UserTargeting dataset with CNN and MLP architectures across varying numbers of labeled examples (300–1,000) and positive ratios in unlabeled data (0.3–0.7), policyPU consistently outperforms state-of-the-art baselines including nnPU, RankPruning, and PMPU—for instance, the Weighter variant achieves 0.975 ROC AUC and 0.916 accuracy on MNIST with only 300 labeled examples at ρ = 0.3, approaching optimal PN performance (0.976 ROC AUC) without access to ground-truth negative labels—establishing that interactive policy-classifier training yields substantial gains across evaluation metrics, though the Separator variant degrades when the proportion of positives in unlabeled data is high (ρ = 0.7), where the threshold-based reward design produces non-optimal feedback for positives near the decision boundary.

2. Context and Motivation

The Core Problem: Learning Binary Classifiers Without Negative Labels

The fundamental problem this paper addresses is deceptively simple: how do you train a binary classifier when you only have positive examples and a pool of unlabeled data, with no explicitly labeled negatives? This is the Positive and Unlabeled (PU) learning problem, and it is fundamentally different from standard supervised learning—which requires examples from both classes—and from standard semi-supervised learning—where at least a few labeled examples exist for each class.

In PU learning, the training data consists of triplets ⟨x, y, s⟩, where x is a feature vector, y ∈ {0, 1} is the true class label (unknown for most data), and s ∈ {0, 1} indicates whether the example is labeled. Crucially, s = 1 guarantees y = 1 (only positives are labeled), but s = 0 provides no information about y—the unlabeled example could be either a positive that simply wasn't labeled or a genuine negative. This asymmetry creates a unique challenge: the absence of a label does not imply the negative class, and treating unlabeled examples as negatives introduces systematic bias that degrades classifier performance.

To appreciate why this distinction matters, consider the typical naive approach: treat all unlabeled examples as negatives and train a standard supervised classifier. The paper refers to this as biased PU learning, and its poor performance is evident throughout the experiments—for instance, on MNIST with 300 labeled examples and ρ = 0.5 (50% of unlabeled data actually being positive), biased PU achieves only 0.535 accuracy and 0.929 ROC AUC (Table II), far below methods that properly account for the unlabeled data's mixed nature. The bias arises because the classifier is trained to push genuine positives in the unlabeled set toward the negative class, forcing a distorted decision boundary.

Why This Problem Is Important: The Pervasiveness of PU Data in Practice

The paper highlights two canonical scenarios that produce PU data, and these represent broad classes of real-world applications (Section I):

E-commerce and user behavior modeling. An e-commerce platform records users who have clicked on advertisements or purchased items—these are unambiguously positive signals of interest. However, users who didn't click or purchase cannot be assumed to be uninterested; they may simply not have seen the advertisement, may be comparison shopping, or may purchase later. The paper's UserTargeting dataset embodies exactly this scenario: labeled users are those who "responded positively to certain products," while unlabeled users are a random sample of everyone else. Building a classifier that identifies potential customers purely from positive click/purchase data and unlabeled user records is a core business problem in recommendation and targeted advertising.

Medical diagnosis. In diagnostic systems, patients who have received a confirmed diagnosis are natural positives. But undiagnosed patients—those without a positive diagnosis record—cannot be assumed disease-free. They may be undiagnosed because the disease is early-stage, because they haven't been tested, or because their symptoms were attributed to something else. Training a classifier that assumes undiagnosed patients are healthy would systematically miss these unlabeled positives, potentially with life-threatening consequences.

These scenarios share a common structure: labeling is inherently one-sided. It is easy (or at least possible) to confirm positive cases, but impossible or impractical to confidently identify negatives. The paper frames this as a "critical task due to its prevalence in various real-world applications" (Section I), and the breadth of prior work they cite—spanning text classification [4, 5, 6], remote sensing [2], time series classification [7], and ecology [1]—confirms that PU learning is not a niche problem but a recurring pattern across domains.

Beyond these domain-specific applications, there is a deeper significance: PU learning represents a regime where label collection costs are asymmetric. When confirming negatives is expensive (e.g., requiring human review to verify non-interest, or medical testing to rule out disease) but positives are readily available, organizations face a choice: either invest heavily to label negatives and use standard supervised learning, or develop methods that work with what they have. The latter scales better and is often the only practical option.

The Established Framework and Its Fundamental Limitation

Prior to this paper, PU learning had converged on a broadly shared framework, but one with a structural weakness. The paper provides an extensive review in Sections I and IV, and understanding this framework is essential to appreciating what policyPU changes.

The SCAR assumption and its consequences. Most modern PU learning methods operate under the Selected Completely At Random (SCAR) assumption: labeled positives are assumed to be a uniform random sample of all positives, independent of their features. Formally:

$p(s = 1 \mid x, y = 1) = p(s = 1 \mid y = 1) = c$

where c is the label frequency—the constant probability that any given positive example gets labeled. This is a strong but often reasonable assumption: in the e-commerce example, it means the probability of a user clicking an ad (and thus becoming labeled) doesn't depend on their features, conditional on the user actually being interested.

Under SCAR, Elkan and Noto [8] proved a critical relationship that connects labeled and unlabeled data:

$p(y = 1 \mid x) = p(s = 1 \mid x) / c$

In words: the probability that an example is truly positive equals the probability that it is labeled (which a classifier can learn from the s = 1 vs s = 0 distinction) divided by the label frequency c. This was a breakthrough because it showed that PU learning could be reduced to estimating c and then learning a standard classifier to predict p(s = 1 | x).

The two-step pipeline and where it breaks. The equation above implies a natural pipeline: (Step 1) estimate the class prior or label frequency c from the PU data; (Step 2) use this estimate to train a classifier, typically by adjusting the loss function or the classifier's output probabilities. This is exactly what the baseline methods in the paper do:

TIcE [16] + nnPU [24]: TIcE uses a decision tree induction approach to estimate the label frequency c, then passes this estimate to nnPU, a non-negative unbiased risk estimator for deep neural networks. The risk estimator from du Plessis et al. [10] rewrites the standard classification risk $R(g) = \alpha E_p[l(g(x))] + (1-\alpha)E_n[l(-g(x))]$ —which requires negative labels—using the observation that the risk on negatives can be expressed in terms of the risk on unlabeled data minus the risk on labeled positives:

$R(g) = \alpha E_p[l(g(x))] - \alpha E_p[l(-g(x))] + E_u[l(-g(x))]$

This requires the label frequency α to be known or estimated.

KM2 [31] + nnPU [24]: KM2 takes a different approach to prior estimation, embedding distributions into a reproducing kernel Hilbert space and using quadratic programming to estimate the mixture proportion of positives in the unlabeled data. The estimate is then fed into the same nnPU risk estimator.
RankPruning [12]: Rather than estimating a class prior, RankPruning attempts to identify and remove incorrectly labeled examples (positives mislabeled as unlabeled, in the PU context) by ranking examples by their predicted class probabilities and pruning those with low confidence. A weighted classifier is then trained on the cleaned dataset.

The paper identifies two fundamental problems with this pipeline paradigm:

1. The estimation and classification steps are coupled but not jointly optimized. The classifier's performance depends critically on the accuracy of the class prior estimate, but the prior is estimated without any feedback from the classifier about what estimate would actually produce the best downstream classification. As the paper states:

"the resulting classifier is affected by the estimation accuracy of prior and label noise. Moreover, the two-step process is unidirectional, i.e., there is no feedback from the classification to the prior and label noise estimation." (Section I)

In other words, the prior estimator gets no signal about whether its estimate is useful for classification. It optimizes a statistical criterion (e.g., mixture proportion fit), but the classifier would prefer an estimate that yields a good decision boundary—and these are not necessarily the same. An estimate that is slightly wrong statistically but places the classification loss in a flat region might be better than a perfect estimate that lands in a sharp minimum.

2. Class priors and label noise rates are unknown in practice. The paper explicitly notes that "the prior distribution of labels or the possible mislabeled examples in the unlabeled dataset are unknown in typical real-world scenarios" (Section I). The TIcE and KM2 methods attempt to estimate these quantities from data, but estimation introduces error, and this error cascades into the classifier training. The experimental results confirm that this cascade is meaningful: on CIFAR-10 with 1,000 labeled examples and ρ = 0.5, TIcE+nnPU achieves only 0.592 accuracy (Table VII), while KM2+nnPU does better at 0.745—a gap of over 15 percentage points that is attributable entirely to the difference in prior estimation quality, since both use the same classifier training method. The classifier has no way to compensate for or recover from a poor prior estimate.

The Gap: No Joint Optimization of Label Assumptions and Classification

The paper positions itself squarely in response to this gap. The core insight is that what the classifier actually needs from unlabeled data is not a single number (the class prior) but a per-example decision about how each unlabeled instance should be used during training. The class prior is a global aggregate; what matters locally is whether this specific unlabeled example functions more like a positive or a negative for the purpose of learning the decision boundary.

This reframes the problem from estimation to decision-making under uncertainty. The question becomes: given the current state of the classifier, what label assignment for each unlabeled example would most improve the classifier? And then: given the classifier's feedback, what label assignment policy would produce the best classifier? These questions are inherently circular—good assignments produce a good classifier, which in turn provides better feedback for making assignments—and this circularity is exactly what reinforcement learning is designed to handle.

The paper draws an explicit parallel to reinforcement learning for noisy data [18], where an RL agent learns to select clean examples from a noisy dataset. But the PU setting is distinct: there is no access to any negative labels, even noisy ones, so the agent must operate with fundamentally less information. The policy network must simultaneously discover which unlabeled examples are negatives and which are hidden positives—without ever seeing a labeled negative.

Why Prior Methods Don't Close This Gap

Beyond the pipeline limitation, the paper identifies specific weaknesses in existing approaches that motivate the need for a joint learning framework:

Class prior estimation is fragile. TIcE depends on decision tree induction to identify regions of feature space with high positive density, but this assumes the feature space has structure that decision trees can capture. KM2's kernel embedding approach requires choosing a kernel and bandwidth, and the quadratic programming sub-routine can be sensitive to the representation. Both methods require downsampling to ~2,000 instances for tractability (Section V-D), discarding information when datasets are large. On the noisy UserTargeting dataset, both methods struggle—TIcE+nnPU achieves 0.895 accuracy, only marginally better than biased PU at 0.885 (Table VIII).

Unbiased risk estimators are theoretically sound but practically limited. The nnPU risk estimator [24] is derived to be unbiased under the SCAR assumption when the class prior is known. However, the non-negative correction needed to prevent overfitting introduces its own bias, and the method requires a loss function satisfying specific constraints (originally $l(x) + l(-x) = 1$ for the ramp loss), limiting the choice of classifier architecture and training objective.

Rank pruning assumes confident predictions are reliable. RankPruning identifies mislabeled examples by ranking them by predicted probability and pruning low-confidence ones. This works well when the classifier's confidence is calibrated, but on difficult datasets or early in training, many correctly labeled examples may have low confidence and incorrectly labeled ones may have high confidence. On CIFAR-10 with 500 labeled examples and ρ = 0.7, RankPruning achieves only 0.584 accuracy (Table VI), substantially below policyPU's 0.770–0.787.

PMPU relies on a margin assumption that doesn't always hold. PMPU [32] assumes a "large positive margin oracle"—that positive instances are located far from the decision boundary. This is a strong geometric assumption. On the UserTargeting dataset, PMPU collapses to 0.631 accuracy (Table VIII), far worse than biased PU (0.885), because the high-dimensional user behavior data doesn't satisfy the margin assumption. PolicyPU makes no such geometric assumptions about the data distribution.

How policyPU Positions Itself

The paper frames policyPU not as another refinement to the estimation pipeline but as a fundamentally different paradigm: instead of estimate-then-classify, it proposes simultaneously decide-labels-and-classify, where "decide" is an active, adaptive policy that improves as the classifier improves. The two components—policy network π_Θ and classifier F_Φ—are trained in an interactive loop:

The policy network observes feature vectors x (states) and outputs actions a representing how each unlabeled example should be treated in classifier training.
The classifier trains on the labeled positives plus the policy-processed unlabeled data, producing predicted class probabilities $\hat{y} = F_\Phi(x)$ .
These predictions serve as rewards to update the policy via REINFORCE: assignments that lead the classifier to predict higher probabilities for those examples receive higher rewards.

This is a dynamic, bidirectional relationship. Unlike the unidirectional pipeline where estimation happens once and the classifier is stuck with whatever estimate was produced, the policy can change its assignments as the classifier's decision boundary evolves. Early in training, when the classifier is poor, the policy receives weak rewards and makes rough assignments. As the classifier improves, rewards become more informative, and the policy can make more nuanced decisions—which in turn further improves the classifier. This positive feedback loop is the central mechanism that the paper hypothesizes will extract more value from unlabeled data than any fixed estimation approach.

The paper explicitly positions this as an end-to-end solution:

"The dynamic and interactive training between the policy maker and the classifier can exploit the unlabeled data in a more effective manner and yield a significant improvement in terms of classification performance." (Abstract)

And further:

"Our policyPU dynamically adjusts its assumptions to U data after making decisions and receiving rewards from the classifier. Thus, it is able to learn a classifier given a PU dataset in an end-to-end fashion." (Section III-A)

The two variants—Weighter and Separator—represent different philosophies about how to use the policy's output. Weighter adopts the mixture model from Elkan and Noto [8] (each unlabeled example is a positive with weight w and a negative with weight 1−w) but learns the weights per-example through the policy rather than computing them from a globally estimated label frequency. This is a direct upgrade to the two-step approach: the weights are now adaptive and optimized for classification performance rather than statistical fidelity. Separator goes further, making hard binary assignments that effectively convert the PU problem into a standard PN problem, with the policy deciding which unlabeled examples get promoted to labeled positives and which get designated as negatives. This is a bolder strategy—it risks mislabeling—but when the policy is accurate, it enables standard supervised training with clean labels.

The paper doesn't claim that reinforcement learning is categorically superior for PU learning. Rather, it argues that the interactive, adaptive structure—regardless of implementation details—is the key missing ingredient in prior work. The REINFORCE algorithm is a means to that end, chosen because it naturally handles the credit assignment problem: which label assignments led to improvements in the classifier, and which were irrelevant or harmful? The policy gradient provides a principled way to answer that question from the downstream classification signal.

Connecting to the Experimental Validation

The paper's experimental design directly tests whether this interactive paradigm matters. By comparing against methods that estimate class priors once (TIcE+nnPU, KM2+nnPU), prune once (RankPruning), or make margin-based assumptions once (PMPU), the experiments isolate the effect of dynamic, reward-driven adaptation. The observed improvements—for example, the Weighter achieving 0.975 ROC AUC on MNIST with 300 labeled examples at ρ = 0.3 versus 0.956 for PMPU and 0.953 for TIcE+nnPU (Table II)—are evidence that the adaptive loop provides value beyond what better estimation alone could achieve.

The difficult cases are equally informative. When the Separator degrades at ρ = 0.7 (high proportion of positives in unlabeled data), it reveals a limitation of the threshold-based reward design: when many unlabeled positives are near the decision boundary, the classifier struggles to assign them high probabilities, leading the policy to receive misleadingly low rewards for these examples. This failure mode is itself a product of the interactive design—the feedback loop can amplify errors as well as correct them—and understanding it points toward future improvements in reward design rather than a fundamental rejection of the approach.

3. Technical Approach

3.1 Reader Orientation

The paper builds a system called policyPU, which is a joint training framework where a policy network and a binary classifier learn simultaneously from positive and unlabeled data through mutual feedback: the policy network decides how to use each unlabeled training example, and the classifier evaluates those decisions, providing rewards that guide the policy toward better future decisions. This solves the fundamental PU learning bottleneck—that conventional methods estimate class priors or label noise before training the classifier in a one-directional pipeline, without any way for the classifier's needs to influence the estimation—by replacing the fixed pipeline with an adaptive, circular training process where label assumptions and classification quality improve together.

3.2 Big-Picture Architecture (Diagram in Words)

The policyPU framework consists of two trainable neural networks that interact through a reward loop within a standard mini-batch training procedure, plus a reward computation module that converts classifier outputs into policy training signals:

1. Policy Network (π_Θ) — The "agent" in reinforcement learning terms. It takes feature vectors x as input (states) and outputs actions a representing how each unlabeled example should be used during classifier training. In the Weighter variant, a is a continuous value in (0, 1) interpreted as the weight of that example being positive (with 1−a being its weight as negative). In the Separator variant, a is a discrete value ∈ {0, 1} making a hard positive/negative assignment after thresholding at 0.5. The policy network is trained via REINFORCE to maximize the expected reward it receives from the classifier.

2. Classifier (F_Φ) — The "environment" that produces rewards. It is a standard neural network (CNN or MLP) trained on the labeled positives plus the policy-processed unlabeled data. In Weighter, it uses a weighted loss function where each unlabeled example contributes to both the positive and negative loss terms with weights a and 1−a respectively. In Separator, it uses standard cross-entropy on the hard-assigned labels. After each mini-batch update, the classifier predicts class probabilities ŷ = F_Φ(x) for all training examples.

3. Reward Computation Module — Converts classifier predictions into scalar rewards R(x, a) using a coherence-based rule: for labeled positives, the reward is simply ŷ (the predicted probability of being positive); for unlabeled examples, if ŷ exceeds a threshold computed from the minimum positive prediction and the mean of high-confidence unlabeled examples, the reward is ŷ (treating the example as if it were positive), otherwise the reward is 1−ŷ (treating it as if it were negative). Intuitively, the reward is high when the classifier's prediction agrees with how the policy treated the example.

Information flow during one mini-batch: (1) A mini-batch of feature vectors is sampled from the training set. (2) The policy network produces actions a_i for each example x_i. (3) The classifier updates its parameters Φ by minimizing the appropriate loss function using the labeled positives and the policy's actions on unlabeled data. (4) The updated classifier predicts ŷ_i for all examples. (5) Rewards R(x_i, a_i) are computed from ŷ_i. (6) The policy network parameters Θ are updated via the policy gradient using the computed rewards.

Both networks update every mini-batch, creating a tight feedback loop where the classifier learns from the policy's evolving label assumptions and the policy learns from the classifier's evolving ability to discriminate.

3.3 Roadmap for the Deep Dive

First, the reinforcement learning formulation and the policy gradient objective (Equation 3), because this defines what the policy network is trying to optimize and establishes the mathematical framework that governs the entire training process.
Second, the policy network itself—what states, actions, and outputs it produces—since understanding the two action types (continuous weights vs. discrete assignments) is prerequisite to understanding the classifier loss functions.
Third, the reward function (Equations 4–5), because the reward is the only signal the policy receives about whether its decisions are good, and its design is the central mechanism that makes the joint training work.
Fourth, the two classifier variants and their loss functions (Equations 8–9), because the classifiers operationalize the policy's actions into actual training and produce the predictions that become rewards.
Fifth, the iterative training procedure (Algorithm 1), including the pre-training step, the target policy network for variance reduction, and the update schedule, since these practical details are essential for reproducing the results.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is to recast PU learning as a reinforcement learning problem where a policy network and a classifier are jointly optimized through interactive, reward-driven training, replacing the conventional two-step pipeline of prior-estimation-then-classification.

Reinforcement Learning Formulation and Policy Gradient Objective

The paper frames the problem of deciding how to use each unlabeled example during classifier training as a reinforcement learning task. This framing is the conceptual backbone of the entire approach, so understanding it clearly is essential.

States, actions, and trajectories. In standard reinforcement learning, an agent observes states, takes actions, receives rewards, and learns a policy that maps states to actions to maximize cumulative reward. The paper maps PU learning onto this structure as follows:

State: The feature vector x of a training example. Each data instance—whether labeled positive or unlabeled—is a state that the policy network observes.
Action: The output a of the policy network for input x. The action represents how this training example will be used in the classifier's loss function. The semantic meaning differs between the two variants: in Weighter, a ∈ (0, 1) is a continuous weight representing the proportion with which this unlabeled example is treated as positive; in Separator, a ∈ {0, 1} is a discrete binary label assignment.
Trajectory: A sequence of mini-batches during training. The paper formulates "a sequence of mini-batches in our training process" as the trajectory, meaning that the policy does not act once and receive a terminal reward, but rather acts repeatedly on every mini-batch and receives instantaneous rewards after each batch.
Reward: A scalar value R(x, a) computed from the classifier's predicted class probability ŷ = F_Φ(x) after the classifier has been updated using the policy's action for that example. The reward design is detailed in the next subsection; for now, the key point is that it measures how well the policy's action "agrees" with what the classifier has learned.

This is not a typical RL setup with long-horizon credit assignment. The policy receives an immediate reward after each mini-batch based on the classifier's current state. There is no discount factor or future reward accumulation—the objective is to maximize the expected reward on the current batch, with the understanding that better policies lead to better classifiers, which in turn provide more informative rewards, creating a virtuous cycle.

The policy gradient objective. The policy network π_Θ with parameters Θ learns a distribution over actions given states: π_Θ(a | x). The objective is to maximize the expected reward over all training examples:

$J(\Theta) = \sum_{x \in P \cup U} \pi_\Theta(a \mid x) R(x, a)$

where $P$ is the set of labeled positive examples, $U$ is the set of unlabeled examples, $\pi_\Theta(a \mid x)$ is the probability (or probability density) of taking action $a$ given state $x$ under the current policy, and $R(x, a)$ is the reward received for taking action $a$ on example $x$ .

What it computes: the expected total reward the policy would receive if it acted on all training examples according to its current action distribution. The sum runs over every example in the training set, and for each example, we weight the reward $R(x, a)$ by how likely the policy is to take action $a$ . The objective $J(\Theta)$ is a function of the policy parameters $\Theta$ because changing $\Theta$ changes the action distribution $\pi_\Theta$ , which changes both which rewards are received and with what probability.

Why this form: this is the standard REINFORCE objective for episodic or step-wise reinforcement learning where the goal is to maximize expected reward. The sum over all training examples treats each example as an independent decision problem rather than modeling dependencies between examples, which is appropriate because the classifier's loss function sums over examples independently. There is no explicit regularization term—the policy is regularized implicitly by the fact that it must produce actions that lead to a good classifier, which in turn produces informative rewards. A policy that outputs extreme or degenerate actions will produce a poor classifier, which will produce uninformative rewards, providing a natural penalty.

REINFORCE update. To actually optimize $J(\Theta)$ , the paper applies the policy gradient theorem, which states that the gradient of the expected reward with respect to the policy parameters can be estimated by sampling actions from the policy and weighting the log-probability gradient by the received reward:

$\nabla_\Theta J(\Theta) = \mathbb{E}_\Theta \left[ \nabla_{\pi_\Theta} \log(\pi_\Theta(a \mid x)) \, R(x, a) \right]$

What it computes: an unbiased estimate of the gradient of $J(\Theta)$ with respect to the policy parameters $\Theta$ . For each example $x$ , we sample an action $a$ from the current policy $\pi_\Theta$ , compute the score function $\nabla_\Theta \log \pi_\Theta(a \mid x)$ (the gradient of the log-probability of the sampled action), multiply it by the scalar reward $R(x, a)$ , and average over all examples. The intuition is that actions that received higher-than-expected rewards get their probabilities increased, while actions that received lower-than-expected rewards get their probabilities decreased.

Why this form: REINFORCE is the simplest policy gradient algorithm and is appropriate here because the action space is low-dimensional (a single scalar per example) and the reward is immediate. More sophisticated algorithms like actor-critic would require learning a value function, adding complexity without clear benefit since there is no temporal structure to the rewards. The score function estimator $\nabla_\Theta \log \pi_\Theta(a \mid x)$ is standard because it provides an unbiased gradient estimate without requiring the reward function to be differentiable with respect to the policy parameters—a critical property since the reward depends on the classifier's predictions, and backpropagating through the classifier into the policy would be both computationally expensive and would create a non-stationary optimization landscape.

In practice, with a mini-batch of size m, the parameter update is:

$\Theta \leftarrow \Theta + \eta \frac{1}{m} \sum_{i=1}^m \nabla_{\pi_\Theta} \log(\pi_\Theta(a_i \mid x_i)) \, R(x_i, a_i)$

where $\eta$ is the learning rate, $m$ is the mini-batch size, and the sum runs over the m examples in the current mini-batch. This is a stochastic gradient ascent step that increases $J(\Theta)$ in expectation.

Policy Network Architecture and Action Types

The policy network π_Θ is a neural network that takes a feature vector x as input and outputs an action a. The paper uses two distinct output formulations depending on whether the Weighter or Separator variant is being used.

Network architecture choices. The paper deliberately uses a shallower architecture for the policy network compared to the classifier. For MNIST and CIFAR-10, the classifier is a 6-layer CNN with 3 convolutional layers ([d-C(3×3,96)-C(3×3,192)-C(1×1,10)-100-1]), while the policy network is a 5-layer CNN with 2 convolutional layers ([d-C(3×3,96)-C(3×3,10)-100-1]). For the UserTargeting dataset, the classifier is a 6-layer MLP ([d-100-50-50-30-1]), paired with either a 4-layer or 6-layer MLP policy network. The paper explains this design choice explicitly:

"For policy networks, we deliberately use a slightly shallower architecture compared to the corresponding targeted classifier. The policy is expected to make rough assumptions at the beginning so that it can gradually adjust itself towards the direction of greater cumulative reward." (Section V-C)

The reasoning is that a shallower network has less capacity and will initially produce coarser, less confident decisions—this is desirable because early in training, when the classifier is poor, overly confident policy decisions would lock in bad assignments before the classifier has a chance to improve. The shallower architecture acts as an implicit form of exploration, forcing the policy to start with rough heuristics and refine them as rewards become more informative.

Continuous actions for the Weighter. In the Weighter variant, the policy network's output layer uses a sigmoid activation to produce $a \in (0, 1)$ , a continuous scalar. This value is interpreted directly as the weight $w$ in the classifier's loss function—the unlabeled example contributes to the positive-class loss with weight $w$ and to the negative-class loss with weight $1-w$ . Because actions are continuous, the policy models a probability density over (0, 1) rather than a probability mass function. In practice, the paper implements this by having the policy network output the parameters of a distribution (typically a Beta distribution, though this detail is not explicitly specified) from which actions are sampled. The sampling step is crucial: it introduces stochasticity that enables exploration of different weight assignments, and the REINFORCE update adjusts the distribution parameters based on which sampled weights produced good rewards.

Discrete actions for the Separator. In the Separator variant, the policy network also outputs a continuous value with sigmoid activation, but this value is thresholded at 0.5 to produce a binary assignment: if $a > 0.5$ , the unlabeled example is assigned as positive; otherwise, it is assigned as negative. The paper describes this as:

"The hard assignment can be seen as the soft labels thresholded with a value, set to 0.5 in our experiments." (Section III-D)

The policy network still models a continuous distribution (so that REINFORCE can compute meaningful gradients through the log-probability), but the action that reaches the classifier is discretized. The reward, however, is computed based on the discrete action—the classifier only sees a binary label, not the underlying continuous value. This creates a subtle challenge: the policy gradient must credit the continuous action distribution for producing a discrete outcome that led to a good reward. The REINFORCE estimator handles this naturally because it only requires the log-probability of the sampled action (which is continuous) and the reward (which depends on the discretized outcome).

Why two variants? The Weighter is the more conservative approach: it never fully commits to a label for any unlabeled example, instead allowing each example to contribute partially to both classes. This is theoretically grounded in Elkan and Noto's observation [8] that each unlabeled example can be viewed as a mixture of positive and negative. It is also more robust to policy errors—if the policy assigns weight 0.6 to an example that is actually positive, that example still contributes substantially to the positive loss, just not with full weight. The Separator is more aggressive: it forces the policy to make binary decisions, essentially converting the PU problem into a standard PN problem with the policy's best guess at the labels. This can work well when the policy is accurate, but it risks mislabeling examples and propagating errors through the classifier. The Separator's hard assignments also make the training more interpretable—one can directly inspect which unlabeled examples the policy assigned as positive versus negative.

Sampling from a target policy network. To reduce the variance of the REINFORCE gradient estimator, the paper uses a target policy network $\pi_{\Theta'}$ that is a delayed copy of the main policy network. Actions are sampled from the target policy rather than the current policy:

"To reduce high variance of the returned reward, we adopt a target policy network for sampling actions. The network is updated every k epochs in our experiment." (Section III-E)

The target policy parameters $\Theta'$ are updated to match $\Theta$ every $k$ epochs (specifically, $k = 3$ in the experiments). This is analogous to the target network technique in DQN: by keeping the action-sampling distribution fixed for several epochs, the policy gradient estimates have lower variance because they are not chasing a constantly shifting distribution. The REINFORCE update still uses the current policy's score function $\nabla_\Theta \log \pi_\Theta(a \mid x)$ , but the actions $a$ are drawn from the frozen target distribution $\pi_{\Theta'}$ , making the reward samples more stable.

Classification Coherence Rewards

The reward function is the only signal the policy network receives about the quality of its decisions. Its design is therefore the most critical component of the framework—a poorly designed reward would train the policy to make decisions that don't actually help the classifier. The paper proposes a coherence-based reward grounded in an intuitive principle: a good policy's assignments should agree with what the classifier eventually learns.

The core intuition. The paper states this intuition directly:

"The intuition behind our reward design is that eventually a good policy will be coherent with the classifier, and this coherence is valid for all data instances and across mini-batches in our framework setting." (Section III-C)

In other words, if the policy assigns an unlabeled example as positive (or gives it high positive weight), and the classifier—after being trained on that assignment—predicts a high probability of being positive for that example, then the assignment was good (the classifier agrees). Conversely, if the classifier predicts a low probability of being positive for an example the policy treated as positive, the assignment was likely wrong (the classifier disagrees). The reward encodes this agreement/disagreement as a scalar that the policy can optimize.

The reward function. Formally, given a training example with feature vector x, an action a from the policy, and the classifier's predicted positive-class probability $\hat{y} = F_\Phi(x)$ , the reward is:

$R(x, a) = \begin{cases} \hat{y}, & \text{if } x \in P, \\ \hat{y}, & \text{if } x \in U \text{ and } \hat{y} \geq \text{threshold}, \\ 1 - \hat{y}, & \text{if } x \in U \text{ and } \hat{y} < \text{threshold}, \end{cases}$

where $P$ is the set of labeled positive examples, $U$ is the set of unlabeled examples, $\hat{y} = F_\Phi(x) \in [0, 1]$ is the classifier's predicted probability that $x$ belongs to the positive class, and $\text{threshold}$ is a scalar reference value computed dynamically from the current mini-batch.

What it computes: the reward is a measure of how "positive-like" the classifier considers example $x$ to be, adjusted by whether the example is formally labeled positive or falls above/below the threshold.

For labeled positives ( $x \in P$ ), the reward is simply $\hat{y}$ : the higher the classifier's confidence that the labeled positive is indeed positive, the higher the reward. This makes intuitive sense—a good classifier should assign high probabilities to known positives—and it provides a stable reference signal for the policy.
For unlabeled examples with high classifier confidence ( $x \in U$ and $\hat{y} \geq \text{threshold}$ ), the reward is also $\hat{y}$ . These are unlabeled examples that the classifier currently believes are positive with sufficient confidence. They are treated analogously to labeled positives in the reward calculation.
For unlabeled examples with low classifier confidence ( $x \in U$ and $\hat{y} < \text{threshold}$ ), the reward is $1 - \hat{y}$ . These are examples the classifier believes are negative. The reward inverts the probability: if the classifier is very confident the example is negative ( $\hat{y} \approx 0$ ), then $1 - \hat{y} \approx 1$ , giving a high reward. This makes sense because the policy likely assigned this example as negative (or gave it low positive weight), and the classifier agrees with that assessment.

Why this form: the reward function does not explicitly depend on the action $a$ —it depends only on $x$ and the classifier's output $\hat{y}$ . This is a deliberate design choice that makes the reward a measure of classifier coherence rather than action correctness. The policy learns to take actions that lead the classifier to produce predictions consistent with those actions, which in turn produces high rewards. This is different from providing a reward based on whether the action matched the (unknown) true label—such a reward would require ground truth, which is unavailable in PU learning. By making the reward depend only on classifier outputs, the framework is self-contained: it uses the classifier's evolving knowledge to teach the policy to make decisions that help the classifier improve further.

The threshold computation. The threshold acts as a decision boundary that separates unlabeled examples into those the classifier considers "positive-like" and those it considers "negative-like." The paper computes it dynamically from the current mini-batch in two steps:

First, compute $\text{thresh}_{\min}$ , the minimum predicted probability among all labeled positive examples in the current mini-batch:

$\text{thresh}_{\min} = \min_{x \in P} F_\Phi(\hat{y} \mid x)$

This represents the lowest confidence the classifier has in any known positive. It establishes a floor: any unlabeled example with predicted probability below this value is less "positive-like" than the least confident labeled positive.

Second, compute the threshold as the mean predicted probability over all labeled positives and those unlabeled examples whose predicted probability exceeds $\text{thresh}_{\min}$ :

$\text{threshold} = \mathbb{E}\left[ \sum_{x \in P \cup U'} F_\Phi(\hat{y} \mid x) \right]$

where $U' = \{ x \mid F_\Phi(\hat{y} \mid x) \geq \text{thresh}_{\min} \}$ is the set of unlabeled examples with predicted probability at or above the minimum-positive threshold.

What it computes: the threshold is the average predicted probability of all examples the classifier is reasonably confident are positive—both the known labeled positives and the unlabeled examples that look sufficiently positive-like. It adapts dynamically to the classifier's current state: if the classifier is poorly trained and gives low probabilities to everything, the threshold will be low, making it easier for unlabeled examples to be treated as positives in the reward. As the classifier improves and becomes more confident about positives, the threshold rises, making the reward function more stringent.

Why this form: the two-step computation ensures the threshold is grounded in the classifier's actual behavior on known positives while also incorporating information from high-confidence unlabeled examples. Using only labeled positives to compute the threshold would be problematic when there are very few labeled examples—the threshold would be based on a small, potentially unrepresentative sample. Including high-confidence unlabeled positives expands the reference set and makes the threshold more stable. The $\text{thresh}_{\min}$ filter ensures that only unlabeled examples the classifier is at least as confident about as its least-confident labeled positive are included, preventing low-confidence unlabeled examples from dragging the threshold down.

Concrete example of the reward in action. Suppose the labeled positives in a mini-batch have predicted probabilities $\{0.85, 0.72, 0.91\}$ , so $\text{thresh}_{\min} = 0.72$ . The unlabeled examples with $\hat{y} \geq 0.72$ are $\{x_1: 0.78, x_2: 0.94\}$ . The threshold is then $(0.85 + 0.72 + 0.91 + 0.78 + 0.94) / 5 = 0.84$ . An unlabeled example with $\hat{y} = 0.88$ gets reward 0.88 (it exceeds the threshold, treated as positive-like). An unlabeled example with $\hat{y} = 0.65$ gets reward $1 - 0.65 = 0.35$ (it's below the threshold, treated as negative-like, and the classifier is moderately confident it's negative). An unlabeled example with $\hat{y} = 0.10$ gets reward $1 - 0.10 = 0.90$ (the classifier is confident it's negative, so the reward is high).

The Separator-specific reward challenge. The paper identifies a limitation of this threshold-based reward when applied to the Separator variant, particularly at high values of ρ (the proportion of positives in unlabeled data):

"it is also possible to increase the threshold if many positives are far from negatives in the unlabeled dataset. As a result, the policy may get non-optimal reward from those positives in U data near the decision boundary due to the threshold setting." (Section V-D)

When ρ is high (e.g., 0.7, meaning 70% of unlabeled examples are actually positive), there are many true positives in the unlabeled set. If most of these are easily classified (far from the decision boundary), they will have high $\hat{y}$ , driving the threshold upward. But some true positives will be near the decision boundary with moderate $\hat{y}$ just below the now-elevated threshold. These examples will be treated as negative-like in the reward ( $R = 1 - \hat{y}$ ), even though the policy correctly assigned them as positive. The policy receives a relatively low reward for a correct decision, creating a perverse incentive. The Weighter variant is less affected because it never fully commits to binary labels—the weights provide a more nuanced signal.

Classifier Architectures and Loss Functions

The classifier F_Φ is a standard neural network (CNN for image data, MLP for tabular data) trained on the labeled positives plus the unlabeled data processed according to the policy's actions. The paper uses two different loss functions corresponding to the two policy variants.

Weighter classifier loss function. In the Weighter variant, the policy network outputs continuous weights $w \in (0, 1)$ for each unlabeled example. The classifier is trained by minimizing:

$L(\Phi) = -\mathbb{E}_\Phi \left[ \sum_{x \in P} \log(F_\Phi(\hat{y} \mid x)) \right] - \mathbb{E}_\Phi \left[ \sum_{x \in U} \left( w \log(F_\Phi(\hat{y} \mid x)) + (1 - w) \log(1 - F_\Phi(\hat{y} \mid x)) \right) \right]$

where $F_\Phi(\hat{y} \mid x)$ is the classifier's predicted probability that $x$ belongs to the positive class, $w = \pi_\Theta(x)$ is the policy network's output weight for unlabeled example $x$ (used directly as the action value), and $\mathbb{E}_\Phi$ denotes the empirical expectation over the mini-batch.

What it computes: the sum of two cross-entropy terms. The first term is the standard cross-entropy loss for labeled positives: each labeled positive contributes $-\log(\hat{y})$ , penalizing the classifier for assigning low probability to known positives. The second term is a weighted combination of positive and negative cross-entropy for each unlabeled example. For an unlabeled example with weight $w$ , the classifier incurs loss $-w \log(\hat{y})$ for predicting it as negative and $-(1-w) \log(1-\hat{y})$ for predicting it as positive. When $w = 1$ , the unlabeled example is treated entirely as a positive (only the first term contributes, same as labeled positives). When $w = 0$ , it is treated entirely as a negative (only the second term). When $w = 0.5$ , the loss is exactly half of each, making the classifier equally penalized for predicting either class—this effectively treats the example as maximally ambiguous.

Why this form: this is a direct implementation of Elkan and Noto's insight [8] that each unlabeled example can be treated as a mixture of a positive with weight $w(x)$ and a negative with weight $1 - w(x)$ . The original formulation computed $w(x)$ from the globally estimated label frequency $c$ and the classifier's own output via $w(x) = \frac{1-c}{c} \cdot \frac{p(s=1 \mid x)}{1 - p(s=1 \mid x)}$ , which makes the weights depend on a single global parameter $c$ and the classifier's current predictions. PolicyPU replaces this with per-example weights learned by the policy network and optimized via reinforcement learning. The key advantage is that the weights are now learned for classification utility rather than derived from a statistical assumption: the policy can assign weight 0.9 to an unlabeled example that is helpful for shaping the decision boundary, even if its true probability of being positive is 0.5, because treating it as mostly positive produces a better classifier.

Separator classifier loss function. In the Separator variant, the policy network's continuous output is thresholded at 0.5 to produce a binary assignment. Unlabeled examples assigned as positive form the set $P'$ ; those assigned as negative form $N'$ . The classifier is trained with standard cross-entropy:

$L(\Phi) = -\mathbb{E}_\Phi \left[ \sum_{x \in P \cup P'} \log(F_\Phi(\hat{y} \mid x)) + \sum_{x \in N'} \log(1 - F_\Phi(\hat{y} \mid x)) \right]$

What it computes: the standard binary cross-entropy loss for supervised learning, where $P \cup P'$ are treated as the positive class and $N'$ as the negative class. Labeled positives and policy-assigned positives are pooled together and contribute $-\log(\hat{y})$ ; policy-assigned negatives contribute $-\log(1-\hat{y})$ . There are no weights or mixing—each example contributes to exactly one loss term with full weight.

Why this form: this is the simplest possible classifier loss and corresponds to the standard supervised learning setup. If the policy makes perfect assignments, this reduces exactly to PN learning (the optimal PN baseline). The Separator thus embodies a bolder hypothesis: that the policy can learn to separate positives from negatives in the unlabeled set accurately enough that standard supervised training on the resulting labeled dataset outperforms more conservative approaches that never fully commit. The risk is that misassigned examples—positives sent to $N'$ or negatives sent to $P'$ —directly corrupt the training signal with full weight.

The implicit regularization of Separator. The paper observes an interesting phenomenon with Separator:

"It is also recognized from the accuracy curve comparison that, our proposal sometimes even yields higher accuracy than the classifier trained on fully labeled PN data with the same parameter setting. We believe that if the true positive and negative examples in U dataset overlap near the decision boundary, the instance weights and even hard assignment given by the policy on these data may serve as an effective regularizer for the classifier." (Section V-E)

This is a non-obvious benefit: when the policy makes errors on ambiguous examples near the decision boundary, those errors may act as label noise that regularizes the classifier, preventing overfitting to the exact positions of borderline examples. This is analogous to the regularizing effect of label smoothing or mixup, where deliberately perturbing labels can improve generalization. The policy introduces a form of data-dependent regularization that is tuned to the classifier's current state, since the policy and classifier co-evolve.

Architecture details. The classifiers for MNIST and CIFAR-10 are 6-layer CNNs with architecture [d-C(3×3,96)-C(3×3,192)-C(1×1,10)-100-1], meaning: dropout, then a 3×3 convolutional layer with 96 filters, then another 3×3 convolutional layer with 192 filters, then a 1×1 convolutional layer with 10 filters, then a fully connected layer with 100 units, then a final single-unit output layer. The classifier for UserTargeting is a 6-layer MLP with architecture [d-100-50-50-30-1]: dropout, then fully connected layers with 100, 50, 50, 30, and 1 units respectively. All networks use ReLU activations, batch normalization, and weight decay.

Training hyperparameters. The paper specifies: Adam optimizer, batch size 128, learning rate fixed to 1e-5. Weight decay is set to 2.0 for classifiers on MNIST and CIFAR-10, and also 2.0 for classifiers on UserTargeting. Policy networks use different weight decay values: for MNIST, Weighter policy uses 2.0 and Separator policy uses 0.5; for CIFAR-10, policies use 0.005 and 1.0 respectively; for UserTargeting, policies use 1e-4. The paper does not justify these specific values beyond stating they were used in experiments—they are presumably tuned hyperparameters.

Pre-Training and Iterative Training Procedure

The interactive training between the policy network and classifier is delicate: if both networks start from random initialization, the classifier provides meaningless rewards to the policy, which in turn makes random label assignments to the classifier, creating a destructive feedback loop where neither network improves. The paper addresses this with a two-phase initialization.

Pre-training the classifier (Phase 1). Before the interactive training begins, the classifier is pre-trained for a few epochs using the naive biased PU approach: all unlabeled examples are treated as negatives, and the classifier is trained with standard cross-entropy on positives (labeled) versus negatives (unlabeled treated as such). The paper states:

"First, the classifier is trained using P and U data directly via a few epochs, then the policy network is also pre-trained with several iterations using the prediction outcome of the classifier." (Section III-E)

On MNIST and CIFAR-10, this pre-training runs for 5 epochs. The resulting classifier is far from optimal (biased PU achieves poor accuracy, e.g., 0.535 on MNIST with 500 labeled examples and ρ = 0.5 in Table III), but it provides a reasonable starting point—the classifier has learned some discriminative features, and its predictions, while biased, are better than random.

Pre-training the policy (Phase 2). Using the pre-trained classifier's predictions, the policy network is then pre-trained for several iterations. The paper does not specify the exact number of pre-training iterations for the policy, but the principle is clear: the policy learns to imitate what the (biased) classifier considers positive versus negative, producing initial label assignments that are at least consistent with the classifier's current knowledge. This ensures that when the interactive loop begins, the policy is not making purely random decisions.

The interactive training loop (Phase 3). After pre-training, the full interactive training proceeds for n epochs. For MNIST and CIFAR-10, n = 300 epochs; for UserTargeting, n = 1,000 epochs. Within each epoch, the data is randomly shuffled and divided into mini-batches of size m = 128. For each mini-batch, the following sequence executes:

Action sampling: For each example $x_i$ in the mini-batch, sample an action $a_i$ from the target policy network $\pi_{\Theta'}$ (the delayed copy), not from the current policy $\pi_\Theta$ . This produces pairs $\{x_i, a_i\}$ for all examples.
Classifier update: Minimize the classifier loss function (Equation 8 for Weighter or Equation 9 for Separator) using the mini-batch data and the sampled actions. For labeled positives, the actions are ignored (they are always treated as positives with weight 1 in Weighter or as P in Separator). For unlabeled examples, the actions determine how they contribute to the loss. The classifier parameters $\Phi$ are updated via one gradient step.
Prediction: The updated classifier computes predicted positive-class probabilities $\hat{y}_i = F_\Phi(x_i)$ for all examples in the mini-batch.
Threshold computation: Calculate $\text{thresh}_{\min}$ as the minimum $\hat{y}$ among labeled positives in the mini-batch. Identify $U'$ , the unlabeled examples with $\hat{y} \geq \text{thresh}_{\min}$ . Compute the threshold as the mean of $\hat{y}$ over $P \cup U'$ (Equation 5).
Reward computation: For each example, compute $R(x_i, a_i)$ using Equation 4 and the computed threshold.
Policy update: Update the policy parameters $\Theta$ using the REINFORCE update (Equation 7) with the computed rewards.
Target policy update (periodic): Every $k$ epochs (where $k = 3$ in the experiments), update the target policy parameters to match the current policy: $\Theta' \leftarrow \Theta$ .

This loop creates an online, interleaved training process. The classifier and policy each take one gradient step per mini-batch, responding to each other's most recent state. This is in contrast to offline or alternating optimization where one component is fully trained before the other is updated. The online nature means the policy must adapt to a non-stationary environment—the classifier is constantly changing, so the optimal policy is a moving target.

The entire procedure in pseudocode (Algorithm 1). The paper formalizes this in Algorithm 1, which captures all the steps above. Key parameters are: $\Theta$ (policy parameters), $\Phi$ (classifier parameters), batch size m, n epochs, and policy update frequency k.

Why this training procedure works. The pre-training phase prevents the "cold start" problem where both networks are random and reinforce each other's errors. The target policy network reduces gradient variance, making REINFORCE more stable. The per-mini-batch updates keep the two networks tightly coupled, allowing the policy to respond quickly as the classifier's decision boundary shifts. The threshold adapts to the classifier's evolving confidence, providing appropriately calibrated rewards throughout training. And the periodic target network updates (every 3 epochs) prevent the policy from chasing its own tail—by keeping the action distribution fixed for several epochs, the reward signal stabilizes.

Convergence behavior observed in experiments. The paper describes an interesting dynamic visible in the learning curves:

"For some scenarios, the policy network seems to be making inaccurate decisions for unlabeled examples at the beginning, yet quickly corrects itself after a few trials. It, in fact, reveals that the proposed interactive learning between the policy network and the classifier is effective for learning on PU datasets." (Section V-E)

This observed "dip and recovery" pattern—where the policy initially degrades from its pre-trained state before improving—is characteristic of reinforcement learning systems exploring their action space. Early exploration leads to some bad decisions, which temporarily hurt the classifier, but the resulting low rewards teach the policy to avoid those bad actions, leading to improved performance. This self-correcting behavior is not possible in the pipeline approach, where a bad prior estimate permanently handicaps the classifier with no mechanism for recovery.

Implementation details. The entire framework is implemented in Chainer. The paper uses ReLU activations, batch normalization, and weight decay throughout. The optimizer is Adam with a fixed learning rate of 1e-5. For TIcE and KM2 baseline methods, the paper downsamples to 2,000 training instances when the dataset exceeds this size, following the defaults in those methods' implementations, which is necessary for their computational tractability but discards information that policyPU can use.

Summary of Design Choices and Their Justifications

Reinforcement learning over supervised estimation: The RL formulation enables the policy to learn from the downstream classifier's performance rather than from statistical fidelity to the data distribution. This aligns the policy's objective with what actually matters—classification accuracy—rather than with an intermediate quantity (class prior) that may not correlate perfectly with classification performance.
Coherence-based reward over ground-truth reward: Since true labels for unlabeled data are unavailable, the reward must come from the classifier itself. The coherence design ensures that the reward is always computable during training and that optimizing it encourages the policy to make assignments the classifier can learn from, rather than requiring oracle knowledge.
Continuous vs. discrete actions as two operating points: The Weighter represents caution (never fully commit, hedge with weights), while the Separator represents boldness (make hard decisions, trust the policy). Having both allows practitioners to choose based on their tolerance for risk and the difficulty of their dataset. The experimental results show Weighter is generally more robust, while Separator can excel when conditions are favorable.
Shallower policy network architecture: Deliberately limiting the policy's capacity forces it to learn coarse patterns first, preventing premature convergence to bad local optima before the classifier provides informative rewards. It also reduces computational cost, since the policy processes every example in every mini-batch.
Target policy network with periodic updates: This variance-reduction technique from the DRL literature (DQN-style target networks) stabilizes REINFORCE training by keeping the action distribution fixed for k = 3 epochs, preventing the gradient estimates from becoming excessively noisy due to the constantly shifting sampling distribution.
Dynamic threshold based on labeled positives and high-confidence unlabeled examples: Grounding the threshold in labeled positives (the only examples with known labels) ensures the reward function respects what the classifier should be learning, while including high-confidence unlabeled positives adapts the threshold to the data distribution and makes it robust when labeled positives are few.
Pre-training both networks before interactive training: This addresses the fundamental bootstrapping problem: RL requires a reasonably informative reward signal to learn, but the reward signal requires a reasonably trained classifier, which requires reasonably good policy decisions. Pre-training breaks this circular dependency by providing an initial (biased but non-random) starting point for both networks.

4. Key Insights and Innovations

Innovation 1: Reframing PU Learning as an Interactive Decision Problem Rather Than a Statistical Estimation Problem

The dominant intellectual paradigm in PU learning prior to this paper was estimation: first estimate some unknown quantity—the class prior α, the label frequency c, or the noise rate—from the positive and unlabeled data, and then use that estimate to train a classifier through a corrected loss function or calibrated output probabilities. This paradigm traces directly to Elkan and Noto [8], whose proof that $p(y=1|x) = p(s=1|x)/c$ under the SCAR assumption established that PU learning reduces to estimating $c$ and learning a standard classifier for $p(s=1|x)$ . Subsequent work refined this core idea: TIcE [16] used decision trees to estimate $c$ , KM2 [31] used kernel mean embeddings, and the unbiased risk estimator family [10, 24] formalized the classifier training step given a known or estimated prior. Even RankPruning [12], which doesn't explicitly estimate a class prior, still follows the estimation paradigm—it estimates which examples are mislabeled, then trains on the cleaned set.

PolicyPU makes a fundamentally different intellectual move: it reframes PU learning not as "estimate the missing parameter, then classify" but as "decide how to use each unlabeled example, and let those decisions and the classifier improve each other." This shifts the problem from statistical estimation to sequential decision-making under uncertainty. The key distinction is that estimation aims to recover a true but hidden quantity ( $c$ , the class prior), while decision-making aims to produce useful actions (label assignments) that lead to a good downstream outcome (classifier accuracy)—and these two objectives are not equivalent. A statistically accurate estimate of $c$ may not produce the best classifier if, for example, the estimation error happens to distort the loss landscape in a particularly harmful way, or if the SCAR assumption is slightly violated in practice.

The reframing also changes what "good" means for the label-assignment component. In the estimation paradigm, a good prior estimator is one that minimizes estimation error (e.g., mean squared error between estimated and true $c$ ). In the decision-making paradigm, a good policy is one whose assignments maximize the classifier's performance—even if those assignments don't correspond to the "true" labels. The paper's experimental observation that Separator "sometimes even yields higher accuracy than the classifier trained on fully labeled PN data" (Section V-E) is unintelligible within the estimation paradigm—how could incorrect labels produce better results than correct ones?—but makes sense within the decision paradigm: the policy's errors on borderline examples near the decision boundary act as a form of data-dependent regularization that improves generalization, analogous to label smoothing.

This is not an incremental refinement of prior estimation approaches. It is a conceptual reframing that opens an entirely different design space: rather than developing better class prior estimators, one can develop better decision policies, better reward functions, and better interaction protocols between the policy and classifier. The REINFORCE implementation in this paper is one concrete instantiation of this space, but the paradigm itself—that label assumptions should be learned for classification rather than before classification—is the intellectual contribution that future work can build upon with different algorithms.

The evidence that this reframing matters—that the interactive decision paradigm yields genuinely different and better outcomes than the estimation paradigm—is visible across all three datasets. On MNIST with 300 labeled examples and $ρ = 0.3$ , policyPU Weighter achieves 0.916 accuracy (Table II), while TIcE+nnPU (estimation pipeline) achieves 0.860 and KM2+nnPU achieves 0.867—a gap of ~5 percentage points that cannot be closed by better prior estimation alone, since both TIcE and KM2 represent state-of-the-art estimation methods. The fact that the decision-making approach consistently outperforms estimation-based methods across varying labeled set sizes, positive ratios, and datasets (Tables II–VIII) provides converging evidence that the reframing captures something real about what makes PU learning hard—and what a solution should optimize for.

Innovation 2: Self-Supervised Reward Design That Eliminates the Need for Oracle Knowledge

Every learning system that adapts behavior based on feedback requires a reward or loss signal. In PU learning, the fundamental obstacle is that no ground-truth labels exist for unlabeled data, so there is no obvious reward for whether a label assignment was correct. The paper's solution—the classification coherence reward—is intellectually distinctive because it creates a reward signal from the very classifier the policy is trying to help, establishing a closed feedback loop that requires no external supervision beyond the labeled positives.

The reward function (Equation 4) is deceptively simple but embodies a specific design philosophy: the reward for an action is the classifier's prediction confidence aligned with the direction implied by that action. For a labeled positive, the reward is $\hat{y}$ —the classifier's confidence that this example is positive. For an unlabeled example, if the classifier is sufficiently confident it's positive (exceeding the dynamically computed threshold), the reward is $\hat{y}$ ; if the classifier is not confident it's positive, the reward is $1-\hat{y}$ . The policy learns to take actions that lead the classifier to produce high-confidence predictions consistent with those actions.

What makes this reward design intellectually significant—rather than merely clever—is that it eliminates a circular dependency without breaking the circle. In standard reinforcement learning, the environment provides rewards that are (at least in principle) independent of the agent's policy. Here, the "environment" is the classifier, which is itself being shaped by the policy's actions. This creates a potential circularity: the policy influences the classifier, which influences the rewards, which influence the policy. The coherence reward design makes this circularity constructive rather than destructive by aligning the reward with a property that should hold if both policy and classifier are improving: a good classifier should be confident about examples whose assigned labels match their true class. The policy cannot game this reward by, for example, assigning all unlabeled examples as positives, because the classifier would then be trained to push everything toward the positive class, leading to low $\hat{y}$ for actual negatives—and those negatives would receive reward $1-\hat{y}$ which would be high, correctly signaling to the policy that they should not have been assigned as positives.

Compare this to what would be required without a self-supervised reward: either (a) ground-truth labels for unlabeled data to provide an accuracy-based reward, defeating the purpose of PU learning; (b) a fixed heuristic for assigning labels, which would recapitulate the unidirectional pipeline problem; or (c) an external reward model trained on some auxiliary task, introducing a new source of error. The coherence reward requires none of these—it operates entirely within the PU training data and the two networks being trained.

The dynamic threshold computation (Equation 5) is a crucial but subtle component of this innovation. By computing the threshold as the mean prediction over labeled positives plus high-confidence unlabeled examples, rather than using a fixed value or computing it solely from labeled positives, the threshold adapts to both the classifier's evolving confidence and the actual distribution of positives in the unlabeled set. When there are few labeled positives, high-confidence unlabeled positives expand the reference set, preventing the threshold from being determined by a small, potentially noisy sample. When the classifier is uncertain (early in training), the threshold is low, making it easier for unlabeled examples to be treated as positives in the reward and giving the policy room to explore. As the classifier improves, the threshold rises, making the reward more stringent and forcing the policy to be more accurate. This adaptive calibration is essential for the positive feedback loop to work across different phases of training.

The paper's acknowledgment of the Separator's degradation at high $ρ$ (Section V-D)—where the threshold can become elevated due to many easily-classified positives in the unlabeled set, causing boundary-case positives to receive low rewards—is intellectually honest and strengthens the contribution. It identifies a failure mode of the reward design that is not a bug in the implementation but a structural consequence of using a scalar threshold to separate "positive-like" from "negative-like" in a high-dimensional decision space. This points toward future work on richer reward structures (e.g., per-example rewards that don't rely on a global threshold) while validating that the current design works well across a wide range of conditions.

The evidence for this innovation's practical impact is the consistent performance of policyPU across diverse configurations: 27 experimental settings (3 datasets × varying labeled examples and $ρ$ values) where the coherence reward drives policy improvement without access to any negative labels. The verification in Figure 5 that the policy's assignment accuracy on unlabeled data increases during training—despite never receiving ground-truth feedback—is direct evidence that the coherence reward provides a valid learning signal.

Innovation 3: The Concept of Per-Example Adaptive Label Weights Learned for Classification Utility

The Weighter variant introduces a concept that is both theoretically grounded in prior work and practically novel in its implementation: each unlabeled example receives an individualized, continuously-valued weight that determines its contribution to the positive and negative loss terms, and these weights are learned by optimizing for classification performance rather than derived from a global statistical parameter.

The theoretical foundation comes from Elkan and Noto [8], who showed that each unlabeled example can be treated as a combination of a positive example with weight $w(x)$ and a negative example with weight $1-w(x)$ , where $w(x)$ is a function of the label frequency $c$ and the classifier's own prediction $p(s=1|x)$ . However, in that formulation, the weights are deterministic functions of a single global parameter $c$ : once $c$ is estimated, every unlabeled example's weight is mechanically determined by the classifier's output, with no per-example degrees of freedom. Two unlabeled examples that the classifier assigns the same $p(s=1|x)$ probability receive identical weights, even if one is much more informative for the decision boundary than the other. And the weights cannot be adjusted to compensate for estimation error in $c$ or violations of the SCAR assumption.

PolicyPU Weighter breaks this determinism in two ways. First, the weights are produced by a separate policy network that takes the feature vector $x$ as input, meaning weights are a learned function of the features themselves, not just the classifier's current prediction. Two examples with similar $\hat{y}$ can receive different weights if they lie in different regions of feature space where the policy has learned different strategies. Second, the weights are optimized via reinforcement learning to maximize classifier performance, not to satisfy a statistical identity. This means the policy can assign, for example, a weight of 0.9 to an unlabeled example that the classifier is uncertain about ( $\hat{y} \approx 0.6$ ) if treating it strongly as positive helps shape the decision boundary in a useful way, even though the "statistically correct" weight under the SCAR assumption might be closer to 0.6.

This is intellectually significant because it decouples the label-assignment mechanism from the class prior estimation problem entirely. The policy doesn't need to know $c$ ; it learns weights that work for classification directly. This explains why policyPU can succeed on datasets like UserTargeting where TIcE and KM2 struggle to accurately estimate the class prior—the class prior may be hard to estimate, but useful per-example weights may still be learnable from the interaction with the classifier.

The evidence for this innovation's distinct value comes from comparing Weighter against methods that use a global prior estimate for weighting. On CIFAR-10 with 1,000 labeled examples and $ρ = 0.5$ , Weighter achieves 0.853 accuracy (Table VII), while TIcE+nnPU—which uses a global prior to construct weights indirectly—achieves only 0.592. The 26-percentage-point gap cannot be explained by classifier architecture (both use nnPU's risk estimator, which is designed for deep networks) or training procedure; it reflects the advantage of learned, per-example, classification-optimized weights over weights derived from a single global parameter estimate.

Furthermore, the Weighter's robustness compared to the Separator—particularly at high $ρ$ where Separator degrades (Figure 2, bottom row, $ρ = 0.7$ )—demonstrates that continuous, example-specific weights provide a form of graceful degradation that hard binary assignments lack. When the policy is uncertain or the reward signal is noisy, a weight of 0.55 vs. 0.45 has a much smaller impact on the classifier than a hard assignment of 1 vs. 0. The Weighter can express uncertainty through intermediate weights, hedging its bets, while the Separator must commit and risk propagating errors with full force.

Innovation 4: Establishing That Dynamic Joint Training Can Outperform Access to Ground-Truth Negative Labels (in Specific Regimes)

One of the most striking and counterintuitive results in the paper is that policyPU-trained classifiers can sometimes outperform the optimal PN baseline—a classifier trained on the same data but with access to the ground-truth labels for all examples, including negatives. This is reported for the Separator variant on CIFAR-10 (Section V-E):

"our proposal sometimes even yields higher accuracy than the classifier trained on fully labeled PN data with the same parameter setting."

This is not just a statement about PU learning being "almost as good" as fully supervised learning. It is a statement that, under certain conditions, not having access to the true negative labels can actually be beneficial—the policy's imperfect assignments act as a regularizer that improves generalization beyond what clean labels achieve.

To understand why this is intellectually significant, consider the default assumption in machine learning: more label information is always better. The entire field of semi-supervised learning is predicated on the idea that adding unlabeled data to a small labeled set improves performance, but it is taken as axiomatic that fully labeled data (PN learning) is the ceiling. PU learning is typically framed as making the best of a bad situation—we don't have negative labels, so we develop methods to compensate. PolicyPU's result inverts this framing: the process of learning the labels jointly with the classifier can produce a better classifier than being given the labels.

The paper's hypothesis for this phenomenon is well-motivated: "if the true positive and negative examples in U dataset overlap near the decision boundary, the instance weights and even hard assignment given by the policy on these data may serve as an effective regularizer." In other words, examples near the decision boundary are inherently ambiguous—small perturbations in their features could flip their class. A classifier trained on clean labels will try to fit exactly those labels, potentially overfitting to noise in the boundary region. The policy, being imperfect, will mislabel some of these boundary examples, and those mislabelings introduce noise that prevents the classifier from fitting the boundary too precisely. This is analogous to the well-established principle that label noise can act as regularization [35], but with a crucial difference: the noise here is learned and adaptive, not random. The policy's errors are concentrated on the examples where the classifier is most uncertain—exactly the examples where regularization is most needed.

This finding also provides a new perspective on the relationship between label quality and generalization. The conventional view is that label noise is harmful and should be minimized (hence RankPruning's approach of identifying and removing noisy labels). PolicyPU suggests that label noise that is correlated with classifier uncertainty can be beneficial, because it regularizes the classifier in exactly the regions where overfitting is most likely. This is a fundamentally different relationship between label noise and model performance than the one assumed by noise-removal approaches.

The caveat, which the paper does not state explicitly but follows from the results, is that this benefit only occurs when the policy is good enough—its errors must be concentrated on genuinely ambiguous examples, not on clear-cut ones. If the policy makes random errors across the entire feature space, the regularizing benefit disappears and the errors simply degrade performance. The fact that policyPU achieves this regime on CIFAR-10 but not universally suggests that the policy's inductive biases (shallower architecture, coherence reward) naturally steer it toward uncertainty-concentrated errors, which is itself an interesting property worth investigating.

The evidence for this innovation is primarily in the CIFAR-10 results: at 300 labeled examples and $ρ = 0.3$ , the optimal PN baseline achieves 0.919 accuracy, while policyPU Separator achieves 0.835 and Weighter achieves 0.830 (Table V)—so the "outperform" claim does not hold in all settings. However, the learning curves in Figure 3 show trajectories where policyPU variants occasionally rise above the optimal PN curve during training before settling near or slightly below it, confirming the phenomenon exists transiently and in specific configurations. The paper's honesty in reporting this as "sometimes" happening rather than as a consistent advantage lends credibility to the observation.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three datasets: (1) MNIST, converted to binary classification by treating digits {0,2,4,6,8} as positive and {1,3,5,7,9} as negative, using the standard 60,000/10,000 train/test split; (2) CIFAR-10, binarized with {airplane, automobile, ship, truck} as positive and {bird, cat, deer, dog, frog, horse} as negative, using the standard 50,000/10,000 train/test split; and (3) UserTargeting, a proprietary e-commerce dataset of 19,032 users each for train and test, where 4,758 users who responded positively to certain products are labeled and the remaining 14,274 are unlabeled, with 153-dimensional feature vectors derived from 6 months of online activity. The MNIST and CIFAR-10 binarization follows the same protocol as Kiryo et al. [24] to ensure comparability.
Base model(s). The classifiers are 6-layer CNNs for MNIST and CIFAR-10 with architecture [d-C(3×3,96)-C(3×3,192)-C(1×1,10)-100-1] (dropout, three convolutional layers, two fully connected layers), and 6-layer MLPs for UserTargeting with architecture [d-100-50-50-30-1]. The policy networks are deliberately shallower—5-layer CNNs with [d-C(3×3,96)-C(3×3,10)-100-1] for image data and 4-layer or 6-layer MLPs for UserTargeting—justified by the design principle that the policy should start with coarse, rough assumptions and gradually refine them. All networks use ReLU activations, batch normalization, and Adam optimization with a fixed learning rate of 1e-5 and batch size 128.
Metrics. Three complementary metrics are reported: accuracy (fraction of test examples correctly classified), ROC AUC (area under the receiver operating characteristic curve, measuring ranking quality across all thresholds), and PR AUC (area under the precision-recall curve, particularly informative for class imbalance). The paper reports all three metrics in tables (Tables II–VIII) to avoid single-sided evaluation, with accuracy learning curves shown in figures (Figures 2–4) for training dynamics. Answers are graded via standard threshold-at-0.5 classification after sigmoid output.
Baselines. Six baseline methods are compared: (1) biased PU, which naïvely treats all unlabeled examples as negatives and trains a standard supervised classifier; (2) TIcE [16] + nnPU [24], where TIcE estimates the label frequency c via decision tree induction and passes it to nnPU's non-negative unbiased risk estimator for deep network training; (3) KM2 [31] + nnPU [24], where KM2 estimates the mixture proportion via kernel mean embeddings and quadratic programming, then feeds the estimate to the same nnPU risk estimator; (4) RankPruning [12], adapted from its original noisy-label setting to PU learning by identifying and pruning likely mislabeled examples from the unlabeled set before training a weighted classifier; (5) PMPU [32], which relies on a large positive margin oracle assumption that positives lie far from the decision boundary and iteratively re-estimates unlabeled labels with random resampling; and (6) optimal PN, a classifier trained on the fully labeled dataset with ground-truth positive and negative labels, serving as a reference ceiling. The proposed methods are policyPU Separator (hard label assignments via thresholded policy outputs) and policyPU Weighter (continuous weights as soft labels).
Generation budget / compute accounting. The paper measures compute implicitly through the number of training epochs (300 for MNIST and CIFAR-10, 1,000 for UserTargeting) and the mini-batch processing, but there is no explicit FLOPs or generation budget accounting as would be found in an inference-time scaling study. All methods use the same classifier architecture and training hyperparameters (batch size 128, learning rate 1e-5, Adam optimizer), making training cost comparable across methods within each dataset, though TIcE and KM2 require additional pre-processing (downsampling to ~2,000 instances for tractability, class prior estimation) that adds overhead not incurred by policyPU.
Cross-validation / statistical protocol. Each experiment is run 5 times with different random seeds, and the mean performance is reported. For RankPruning, a 5-fold cross-validation is used internally to prune incorrectly labeled examples and compute class weights. PolicyPU uses a two-phase pre-training protocol (5 epochs of biased PU pre-training for the classifier, followed by policy pre-training using classifier predictions) before 300 (or 1,000) epochs of interactive training with policy updates every k = 3 epochs. The target policy network is updated every 3 epochs for variance reduction. No test-set cross-validation is used for hyperparameter selection; the paper reports results on the standard test splits directly. The experimental grid varies two factors independently: the number of labeled examples nl ∈ {300, 500, 1000} for MNIST and CIFAR-10, and the proportion of positives in unlabeled data ρ ∈ {0.3, 0.5, 0.7}, with the number of unlabeled examples fixed at 3 × nl.

Main Quantitative Results

MNIST Results: policyPU Consistently Outperforms Estimation-Based Methods Across Label Budgets and Positive Ratios

The main quantitative picture on MNIST is that policyPU Weighter dominates all baselines across nearly every configuration of labeled data quantity and positive ratio, while policyPU Separator is competitive at low-to-moderate ρ but degrades at high ρ. The full numerical results after 300 epochs are presented in Tables II (nl = 300), III (nl = 500), and IV (nl = 1,000), with accuracy learning curves shown in Figure 2.

Headline numbers at the smallest label budget (nl = 300, Table II). At ρ = 0.3, policyPU Weighter achieves 0.916 accuracy, 0.975 ROC AUC, 0.975 PR AUC, substantially outperforming the best baseline (KM2+nnPU at 0.867 accuracy, 0.951 ROC AUC, 0.954 PR AUC) and approaching the optimal PN ceiling (0.919 accuracy, 0.976 ROC AUC, 0.977 PR AUC). The gap between Weighter and the top estimation baseline is 4.9 percentage points in accuracy—a large margin given that all methods share the same classifier architecture and differ only in how they handle unlabeled data. At ρ = 0.5, Weighter achieves 0.880 accuracy, 0.948 ROC AUC, 0.948 PR AUC, again leading all baselines (best: KM2+nnPU at 0.861 accuracy, 0.938 ROC AUC, 0.943 PR AUC). At ρ = 0.7, Weighter achieves 0.818 accuracy, 0.915 ROC AUC, 0.909 PR AUC, still ahead of the best baseline (TIcE+nnPU at 0.828 accuracy, 0.942 ROC AUC, 0.940 PR AUC on accuracy—though note TIcE+nnPU actually edges ahead on accuracy at this specific configuration, 0.828 vs. 0.818).

The Separator's degradation at high ρ is clearly visible. At ρ = 0.7 with nl = 300, Separator achieves only 0.722 accuracy (0.890 ROC AUC, 0.893 PR AUC)—performing worse than biased PU (0.510 accuracy, 0.874 ROC AUC, 0.865 PR AUC) on accuracy, and substantially below the other PU methods (TIcE+nnPU: 0.828 accuracy). This is consistent with the paper's explanation that the threshold-based reward produces non-optimal feedback when many positives in the unlabeled set are near the decision boundary, causing the policy to receive low rewards for correctly assigned positives. The degradation is systematic across label budgets: at nl = 500 and ρ = 0.7 (Table III), Separator achieves 0.760 accuracy vs. Weighter's 0.833; at nl = 1,000 and ρ = 0.7 (Table IV), Separator achieves 0.743 accuracy vs. Weighter's 0.843.

Kernel-based prior estimation (KM2+nnPU) is the strongest estimation baseline on MNIST. At ρ = 0.5 and nl = 500 (Table III), KM2+nnPU achieves 0.894 accuracy, outperforming TIcE+nnPU (0.794) and RankPruning (0.722). However, KM2+nnPU's advantage over TIcE+nnPU is not consistent—at nl = 300 and ρ = 0.7, TIcE+nnPU (0.828 accuracy) outperforms KM2+nnPU (0.722), indicating that the quality of class prior estimation varies with both the label budget and the data distribution. PolicyPU, by not depending on a class prior estimate at all, avoids this sensitivity.

RankPruning is inconsistent on MNIST. At nl = 300 and ρ = 0.3 (Table II), RankPruning achieves only 0.745 accuracy, far below KM2+nnPU (0.867) and policyPU (0.862–0.916). At nl = 500 and ρ = 0.3 (Table III), it improves to 0.872 but still trails KM2+nnPU (0.892) and policyPU (0.903–0.942). At nl = 1,000 and ρ = 0.7 (Table IV), it achieves 0.853, competitive with Weighter (0.843) and outperforming Separator (0.743). The variability suggests that pruning effectiveness depends on the classifier's calibration quality, which is dataset- and budget-dependent.

PMPU performs well on MNIST but not dominantly. PMPU achieves 0.861 accuracy at nl = 300 and ρ = 0.3 (Table II), comparable to KM2+nnPU (0.867) and policyPU Separator (0.862) but below Weighter (0.916). Its reliance on the large positive margin assumption—that positives are far from the decision boundary—appears to be reasonably satisfied by the MNIST image features, but the assumption's validity doesn't translate into an advantage over methods that don't require it.

The learning curves (Figure 2) reveal training dynamics. The accuracy curves show that policyPU variants sometimes require more epochs to converge than baselines—for instance, at nl = 300 and ρ = 0.5, Weighter's accuracy rises gradually from ~0.75 at epoch 50 to ~0.88 at epoch 300, while KM2+nnPU plateaus earlier. The paper notes that at ρ = 0.7, "it takes longer for both of the classifiers in our framework to start to predict reasonably," reflecting the challenge of the reward function when positives dominate the unlabeled set.

CIFAR-10 Results: policyPU Outperforms Baselines, Sometimes Exceeding Optimal PN

The CIFAR-10 results (Tables V–VII, Figure 3) paint a picture where policyPU variants are the top performers in most configurations, with the striking result that Separator sometimes exceeds the optimal PN baseline.

Headline numbers at nl = 300 (Table V). At ρ = 0.3, policyPU Separator achieves 0.835 accuracy, 0.915 ROC AUC, 0.877 PR AUC, leading all PU methods (KM2+nnPU: 0.814 accuracy, 0.895 ROC AUC, 0.877 PR AUC; Weighter: 0.830 accuracy, 0.915 ROC AUC, 0.880 PR AUC). The optimal PN baseline achieves 0.919 accuracy—so neither PU method approaches the fully supervised ceiling, but both outperform the estimation baselines. At ρ = 0.5, Separator (0.808 accuracy) and Weighter (0.798) again lead, ahead of RankPruning (0.754) and KM2+nnPU (0.736). At ρ = 0.7, Separator (0.750) edges out Weighter (0.741) and PMPU (0.721), though all methods perform relatively poorly at this high positive ratio.

At nl = 500 (Table VI): Separator achieves 0.860 accuracy at ρ = 0.3, Weighter achieves 0.859, both substantially ahead of the best baseline (KM2+nnPU: 0.803). At ρ = 0.5, Separator (0.834) and Weighter (0.823) again lead. At ρ = 0.7, Separator achieves 0.787, outperforming KM2+nnPU (0.744).

At nl = 1,000 (Table VII): Weighter achieves 0.884 accuracy at ρ = 0.3, Separator achieves 0.856. At ρ = 0.5, Weighter (0.853) and Separator (0.846) lead RankPruning (0.712) and the risk-estimator baselines (TIcE+nnPU: 0.592, KM2+nnPU: 0.745). At ρ = 0.7, Separator achieves 0.833, outperforming all baselines.

The "exceeding optimal PN" observation appears in the learning curves (Figure 3). The paper states: "our proposal sometimes even yields higher accuracy than the classifier trained on fully labeled PN data with the same parameter setting." Examining the curves, at nl = 300 and ρ = 0.3, the optimal PN accuracy is approximately 0.92 and plateaus there, while Separator and Weighter both reach approximately 0.83–0.84—so the exceedance is not universal. However, the paper's hypothesis about regularization from policy errors on boundary examples is supported by the fact that policyPU classifiers show better generalization relative to their training data disadvantage than would be expected: they train on substantially less labeled information than optimal PN yet achieve accuracy levels that are proportionally higher than what the label deficit would predict under a linear scaling assumption.

TIcE+nnPU performs poorly on CIFAR-10. At nl = 1,000 and ρ = 0.5 (Table VII), TIcE+nnPU achieves only 0.592 accuracy—barely above biased PU (0.618) and far below KM2+nnPU (0.745). This suggests that decision-tree-based label frequency estimation fails on the more complex CIFAR-10 feature space, where the decision tree cannot reliably identify regions of high positive density. The pipeline approach's vulnerability is starkly illustrated here: a poor prior estimate cascades into a 15+ percentage point accuracy loss compared to a good estimate (KM2+nnPU), with the classifier having no recourse.

RankPruning shows high variance on CIFAR-10. At nl = 500 and ρ = 0.7 (Table VI), RankPruning achieves 0.584 accuracy, worse than biased PU (0.607). At nl = 1,000 and ρ = 0.3 (Table VII), it achieves 0.825, competitive with policyPU (0.856–0.884). The pruning strategy appears highly sensitive to the classifier's calibration and the dataset difficulty, making it an unreliable choice.

UserTargeting Results: policyPU Advances Over Baselines on a Noisy, Real-World Dataset

The UserTargeting dataset differs fundamentally from the image benchmarks: it has no ground-truth negatives (only labeled positives and unlabeled users), higher noise, lower-dimensional features, and a different domain. The results (Table VIII, Figure 4) test whether policyPU's advantages transfer to this more challenging, realistic setting.

Headline numbers (Table VIII, left: 4-layer policy; right: 6-layer policy). With a 4-layer policy network, policyPU Separator achieves 0.974 ROC AUC, 0.937 accuracy, 0.898 PR AUC, substantially outperforming all baselines. The best estimation baseline is KM2+nnPU at 0.957 ROC AUC, 0.896 accuracy, 0.846 PR AUC—a gap of 4.1 percentage points in accuracy. Weighter achieves 0.971 ROC AUC, 0.914 accuracy, 0.885 PR AUC, also ahead of baselines but behind Separator. With a 6-layer policy network, results are similar: Separator achieves 0.972 ROC AUC, 0.941 accuracy, 0.892 PR AUC; Weighter achieves 0.969 ROC AUC, 0.919 accuracy, 0.883 PR AUC.

PMPU collapses on UserTargeting. PMPU achieves only 0.858 ROC AUC and 0.631 accuracy—far worse than even biased PU (0.951 ROC AUC, 0.872 accuracy). The paper attributes this to the noise level: "we speculate that the reason is due to the fact that this user behavior dataset is noisier than MNIST and CIFAR-10. Hence, the calculation of the significant parameter, τ, for PMPU may be severely impacted." This is a telling failure mode: methods that rely on specific geometric or statistical assumptions about the data distribution can break catastrophically when those assumptions are violated, while policyPU's interactive approach makes no such assumptions and degrades more gracefully.

Biased PU is a surprisingly strong baseline on UserTargeting. At 0.951 ROC AUC and 0.872 accuracy (4-layer policy comparison), biased PU outperforms PMPU (0.858/0.631) and RankPruning (0.949/0.880 on accuracy—RankPruning is slightly higher on accuracy but comparable on ROC AUC). This suggests that for this particular problem, the assumption that unlabeled users are mostly negative may be approximately true, making the naive approach reasonable. However, policyPU still substantially outperforms biased PU (0.937 vs. 0.872 accuracy), showing that even when the naive assumption is decent, learning per-example label assignments provides meaningful gains.

Class prior estimation methods (TIcE, KM2) are competitive but not dominant. TIcE+nnPU achieves 0.957 ROC AUC, 0.895 accuracy; KM2+nnPU achieves 0.957 ROC AUC, 0.896 accuracy. Both are strong baselines, outperforming biased PU by ~2.3 percentage points in accuracy, but both are clearly below policyPU Separator (0.937 accuracy). The paper notes that "both class prior estimation methods seem to have difficulty to accurately approximate true values" on this noisy dataset, but they still provide useful signals.

Training takes longer to converge on UserTargeting. The learning curves (Figure 4) show that policyPU methods require more epochs to converge on UserTargeting (1,000 epochs total) compared to the 300-epoch experiments on image data. The paper acknowledges: "We recognize that it takes a bit longer for policy networks to learn consistent policy. We assume it is still because of the high noise level, which makes the policy learning converge slower." The Separator's accuracy rises gradually from ~0.87 at epoch 200 to ~0.937 at epoch 1,000, while Weighter shows a similar gradual climb. This slower convergence is a practical consideration for deployment but does not negate the final performance advantage.

Ablation Studies and Robustness Checks

Policy learning verification (Figure 5): The paper verifies that the policy is genuinely learning to improve its assignment accuracy on unlabeled data by tracking the "correct assignment rate" of the Separator policy during training on MNIST (nl = 300, ρ = 0.3) and CIFAR-10 (nl = 300, ρ = 0.3). On MNIST, the correct assignment rate starts at approximately 0.65 at epoch 0 (after pre-training), dips to approximately 0.62 around epoch 25, then rises steadily to approximately 0.75 by epoch 200. On CIFAR-10, the rate starts at approximately 0.60, dips to approximately 0.52 at epoch 30, then recovers and climbs to approximately 0.68 by epoch 200. This "dip and recovery" pattern confirms the paper's claim that the policy initially makes inaccurate decisions but the interactive training "quickly corrects itself after a few trials." The fact that the assignment rate improves without any ground-truth label feedback—purely from the coherence reward—is the central evidence that the reward design provides a valid learning signal.

Weighter vs. Separator across ρ values (Tables II–VII): The comparison between Weighter and Separator across all configurations serves as an ablation of the action type (continuous soft labels vs. discrete hard assignments). At ρ = 0.3 and 0.5, Separator and Weighter perform comparably or Separator slightly leads (e.g., CIFAR-10, Table V: Separator 0.835 vs. Weighter 0.830 at ρ = 0.3). At ρ = 0.7, Weighter consistently outperforms Separator on MNIST (e.g., Table III: 0.833 vs. 0.760), while on CIFAR-10 Separator sometimes maintains an edge (Table VII: 0.833 vs. 0.816). This interaction between action type and positive ratio confirms that the threshold-based reward creates specific challenges for discrete actions when positives dominate the unlabeled set, but that continuous weights provide robustness.

Policy network depth ablation (Table VIII): On UserTargeting, the paper tests two policy network architectures: a 4-layer MLP and a 6-layer MLP, both paired with the same 6-layer MLP classifier. Separator with the 4-layer policy achieves 0.974 ROC AUC, 0.937 accuracy; with the 6-layer policy, 0.972 ROC AUC, 0.941 accuracy. Weighter with the 4-layer policy achieves 0.971 ROC AUC, 0.914 accuracy; with the 6-layer policy, 0.969 ROC AUC, 0.919 accuracy. The small differences (within ~0.5–0.7 percentage points in accuracy) suggest that the framework is not highly sensitive to policy network depth within this range, though the shallower policy (4-layer) was deliberately designed to make "rough assumptions at the beginning" and the results confirm it performs comparably to the deeper variant.

Number of labeled examples ablation (Tables II, III, IV; V, VI, VII): Across MNIST and CIFAR-10, increasing nl from 300 to 500 to 1,000 consistently improves all methods' performance, as expected. On MNIST at ρ = 0.5, Weighter improves from 0.880 (nl = 300) to 0.923 (nl = 500) to 0.935 (nl = 1,000). The relative advantage of policyPU over baselines is generally stable or slightly increasing with more labels: at nl = 300, Weighter leads the best baseline (KM2+nnPU) by 1.9 percentage points (0.880 vs. 0.861); at nl = 1,000, the lead is 3.8 percentage points (0.935 vs. 0.897). This suggests that policyPU extracts more value from additional labeled data than the estimation-based methods, consistent with the hypothesis that the interactive training makes better use of label information.

Positive ratio in unlabeled data ablation (ρ ∈ {0.3, 0.5, 0.7}): Across all methods and datasets, performance degrades as ρ increases—more positives in the unlabeled set means a harder problem with fewer likely negatives to learn from. On CIFAR-10 with Weighter and nl = 1,000 (Table VII): 0.884 at ρ = 0.3, 0.853 at ρ = 0.5, 0.816 at ρ = 0.7. The degradation is expected and affects all methods, but policyPU's relative advantage is largest at intermediate ρ (0.5): at nl = 1,000 on CIFAR-10, Weighter leads KM2+nnPU by 10.8 percentage points at ρ = 0.5 (0.853 vs. 0.745), compared to 7.3 points at ρ = 0.3 (0.884 vs. 0.811) and 11.3 points at ρ = 0.7 (0.816 vs. 0.703). The Separator's degradation at high ρ is specific to that variant and is discussed above.

Pre-training ablation (implicit in Algorithm 1): The paper does not run an explicit experiment without pre-training, but the algorithm description makes clear that pre-training is essential: "a pre-training step, that simply use unlabeled examples as negative, is applied before the interactive learning to stabilize the process." The learning curves showing dips early in training (Figure 5's dip-and-recovery, Figure 3's Separator trajectories) suggest that even with pre-training, the policy explores and makes early mistakes; without pre-training, both networks would start from random initialization, providing meaningless rewards, and the interactive loop would likely fail to converge entirely. The pre-training is thus a necessary bootstrapping mechanism rather than an optional enhancement.

Target policy update frequency (k = 3 epochs): The paper fixes the target policy update frequency at k = 3 epochs for all experiments. No ablation over different k values is reported. This is a hyperparameter whose sensitivity is untested—values of k = 1 (no target network, highest variance), k = 5, or k = 10 could produce different convergence behaviors, and the choice of k = 3 appears to be a reasonable default without empirical justification in the paper.

Negative results on PMPU for UserTargeting (Table VIII): PMPU's collapse to 0.858 ROC AUC and 0.631 accuracy on UserTargeting is a significant negative result that validates policyPU's design philosophy. PMPU assumes a "large positive margin" where positives are far from the decision boundary, and the UserTargeting dataset's noise level violates this assumption. PolicyPU makes no geometric assumptions and maintains strong performance (0.974 ROC AUC), demonstrating robustness to conditions that break assumption-dependent methods.

Critical Assessment

The experiments demonstrate several clear empirical facts: policyPU, particularly the Weighter variant, consistently outperforms estimation-based PU learning baselines across three datasets, multiple labeled-data budgets, and multiple positive-ratio settings; the coherence-based reward successfully trains a policy network to make increasingly accurate label assignments without ground-truth feedback; and the interactive training paradigm produces classifiers that approach or, in specific configurations on CIFAR-10, temporarily exceed the performance of fully supervised PN training. But there are important boundaries around what these results demonstrate and where the experimental evidence is thinner.

Claim 1: "The classiﬁers learned by our framework yield consistent improvements in terms of ROC AUC, accuracy and PR AUC on three datasets." This claim is well-supported for the datasets and configurations tested. Across the 36 pairwise comparisons in Tables II–VIII (3 metrics × varying nl and ρ), policyPU variants achieve the highest score in the large majority of cases. The consistency across metrics is meaningful—it is not the case that policyPU gains accuracy at the expense of ROC AUC or vice versa. The gains are substantial in magnitude: on MNIST with nl = 300 and ρ = 0.3, Weighter's 0.916 accuracy is 5.6 percentage points above the best baseline (KM2+nnPU at 0.867), and on UserTargeting, Separator's 0.937 accuracy is 4.1 points above KM2+nnPU (0.896). These are practically meaningful improvements.

However, "three datasets" overstates the diversity. MNIST and CIFAR-10 are both image classification benchmarks with clean, structured features, preprocessed identically following [24], and using the same CNN architectures. The UserTargeting dataset adds a tabular, noisy, real-world domain, but even so, all three datasets are binary classification problems with moderate dimensionality. The paper does not test on text data, graph data, time series, or highly imbalanced datasets (the positive ratios tested—0.3, 0.5, 0.7—all represent substantial positive presence). Claims about "consistent improvements" are valid within the tested scope but should not be extrapolated to all PU learning scenarios without further evidence.

Claim 2: "The dynamic and interactive training between the policy maker and the classiﬁer can exploit the unlabeled data in a more effective manner and yield a significant improvement." The evidence for this claim is indirect but convincing. The explicit comparison is between policyPU and methods that use a fixed, one-shot estimation of the class prior (TIcE+nnPU, KM2+nnPU). PolicyPU's superior performance is consistent with the mechanism being the interactive, adaptive nature of the training, but the experiments do not isolate this mechanism. An ablative comparison that would strengthen the claim would be: train the policy network once before classifier training (making it a one-shot label assigner, equivalent to the pipeline approach but with a neural network estimator), and compare this to the full interactive training. If the interactive version substantially outperforms the one-shot version, the dynamic aspect is specifically validated. The paper does not run this ablation, so the possibility remains that policyPU's advantage comes from other factors—the per-example nature of the weights, the neural network policy architecture, or the specific reward design—rather than from the interactivity per se.

Claim 3: "Our proposal sometimes even yields higher accuracy than the classiﬁer trained on fully labeled PN data" (the "exceeds optimal PN" claim). This claim is supported with important caveats. The paper reports this as occurring on CIFAR-10 "sometimes" and attributes it to regularization from policy errors on boundary examples. However, examining the final accuracy numbers in Tables V–VII, policyPU never consistently beats optimal PN at convergence—the optimal PN baseline achieves 0.919, 0.874, 0.879 at nl = 300, 500, 1000 and ρ = 0.3, while policyPU reaches 0.830–0.835, 0.859–0.860, 0.856–0.884 respectively. The "sometimes exceeds" claim likely refers to transient behavior visible in the learning curves (Figure 3) where policyPU trajectories cross above the optimal PN line during training before settling below it. This is an interesting phenomenon—it suggests the policy-induced regularization can temporarily produce better generalization—but it is not a sustained advantage and does not mean policyPU is "better than fully supervised learning" in any practical sense. The claim is presented honestly ("sometimes"), but readers should understand it as a scientific observation about regularization dynamics rather than a practical recommendation to prefer PU learning over PN learning when both are available.

Genuine weaknesses in the experimental design:

No statistical significance testing. The paper reports results averaged over 5 runs, but no standard deviations, confidence intervals, or significance tests are provided. Given the 500-question test sets for MNIST and CIFAR-10, differences of 1–2 percentage points in accuracy may not be statistically significant. The 5-run averaging provides some indication of stability, but without variance estimates, the reliability of small performance differences is unknown.
No held-out validation set for hyperparameter tuning. The paper appears to use the test set directly for evaluation without a separate validation split for selecting hyperparameters like weight decay, learning rate, or the policy update frequency. The weight decay values differ substantially across datasets and between Weighter and Separator (e.g., MNIST Weighter policy: 2.0, Separator policy: 0.5; CIFAR-10 Weighter policy: 0.005, Separator policy: 1.0), suggesting dataset-specific tuning. If this tuning was done by monitoring test-set performance, the reported numbers may be optimistically biased.
UserTargeting has no ground-truth negatives, making evaluation ambiguous. The paper does not explain how accuracy, ROC AUC, and PR AUC are computed for UserTargeting if there are no labeled negatives in the test set. For PU datasets, standard metrics require knowing which unlabeled test examples are actually negative—if this is unavailable, the evaluation methodology needs clarification. The paper may be using the original positive/negative labels from the recommendation outcome, but the dataset description states "this dataset is split 50-50 to train and test. Both of them have 4,758 labeled and 14,274 unlabeled users" with no mention of labeled negatives. This ambiguity undermines confidence in the UserTargeting results.
No comparison to more recent PU learning methods. The baselines are from 2016–2018 (TIcE 2018, KM2 2016, nnPU 2017, RankPruning 2017, PMPU 2018). While these were state-of-the-art at the time of the paper's publication (2019–2020), the paper does not compare against self-supervised or contrastive PU learning methods that were emerging contemporaneously. This is not a fatal flaw—the baselines are appropriate for establishing the interactive paradigm's advantage over the estimation pipeline—but it bounds the "state-of-the-art" claim.
The Separator's degradation at high ρ is acknowledged but not solved. Separator at ρ = 0.7 on MNIST with nl = 300 achieves 0.722 accuracy, worse than TIcE+nnPU (0.828) and KM2+nnPU (0.722, tied). The paper identifies the reward threshold as the culprit but does not propose or test a fix (e.g., an adaptive threshold, per-example reward scaling, or a different reward design for the Separator). This leaves a known failure mode unaddressed.
Limited exploration of the policy network design space. The paper uses one policy architecture per dataset (shallower than the classifier) without ablating width, depth, activation functions, or initialization strategies. The claim that "shallower architecture forces rough initial assumptions" is plausible but untested—a same-depth policy network might converge faster or to a different optimum, and this comparison is not made.
No sensitivity analysis for k (policy update frequency) or the pre-training duration. These hyperparameters are fixed at k = 3 and 5 epochs without justification or ablation. The convergence behavior may be sensitive to these choices, and practitioners seeking to apply policyPU to new datasets would need guidance on setting them.

Missing experiments that would strengthen the paper:

One-shot policy baseline: Train the policy once before classifier training (using the coherence reward with a fixed pre-trained classifier) and then freeze it. Compare to the interactive version to isolate the value of dynamic adaptation.
Varying amounts of unlabeled data: The paper fixes nu = 3 × nl. Testing with nu = nl, nu = 5 × nl, and nu = 10 × nl would reveal how policyPU scales with unlabeled data quantity, a practically important dimension.
Extremely low label regimes: nl = 100 or even nl = 50 for MNIST would test whether the interactive paradigm works when the labeled positive set is very small, where class prior estimation would be most challenging.
Ablation of the threshold in the reward function: Compare the dynamic threshold (Equation 5) to a fixed threshold (e.g., 0.5), to using only labeled positives for the threshold, and to a reward that is simply ŷ for all examples. This would quantify the importance of the threshold design.
Direct measurement of class prior estimation improvement: Track whether the policy's implicit class prior estimate (the fraction of unlabeled examples assigned as positive) converges to the true ρ over training. This would connect the policy learning to the traditional estimation framework.
Comparison to ensemble or multi-training methods: Methods like bagging or co-training that also use iterative, interactive mechanisms are not compared. This would help distinguish policyPU's specific contribution from the general benefit of iterative refinement.

In summary, the experiments convincingly demonstrate that policyPU outperforms estimation-based PU learning methods on the tested benchmarks, and the verification that the policy's assignment accuracy improves without ground-truth labels is strong evidence for the coherence reward's validity. The key conditions for the claims to hold are: (1) the dataset has sufficient structure that a neural network policy can learn useful per-example weights, (2) the labeled positive set is large enough to provide a reference signal for the reward threshold, (3) the positive ratio in unlabeled data is not so high (≫0.7) that the Separator's threshold-based reward breaks down, and (4) pre-training is used to bootstrap the interactive loop. The experiments do not demonstrate that policyPU works without pre-training, that it outperforms fully supervised PN learning in a sustained way, or that it scales favorably in terms of wall-clock training time compared to the simpler estimation baselines—all of which are relevant practical considerations that the paper does not address.

6. Limitations and Trade-offs

The Coherence Reward Can Produce Misleading Feedback When Positive Ratios Are High

The assumption or constraint. The threshold-based reward function in Equation 4 determines whether an unlabeled example is treated as positive-like or negative-like based on its classifier-predicted probability $\hat{y}$ relative to a dynamically computed threshold. The paper acknowledges a specific failure mode: when the proportion of positives in unlabeled data is high, easily classified positives drive the threshold upward, causing true positives near the decision boundary to fall below the threshold and receive reward $1 - \hat{y}$ despite being correctly assigned as positive by the policy. The paper states this explicitly:

"it is also possible to increase the threshold if many positives are far from negatives in the unlabeled dataset. As a result, the policy may get non-optimal reward from those positives in U data near the decision boundary due to the threshold setting." (Section V-D)

The consequence. The policy receives a low reward for a correct decision—assigning a boundary-case positive as positive—because the classifier can't confidently classify it yet, and the elevated threshold places it in the negative-like category. This creates a perverse training signal: the policy is penalized for making accurate assignments on the hardest-to-classify positives, pushing the policy toward mislabeling these examples as negatives. The Separator variant is particularly affected because it must make hard binary decisions, and mislabeling a boundary positive as negative propagates with full weight through the cross-entropy loss (Equation 9). The Weighter is "not impacted as severe as the policyPU separator" because its continuous weights provide a more nuanced signal—an example can contribute partially to both classes, hedging the policy's uncertainty.

What evidence exists in the paper. The degradation is clearly visible in the MNIST results. At nl = 300 and ρ = 0.7 (Table II), policyPU Separator achieves only 0.722 accuracy, substantially below Weighter (0.818) and even below TIcE+nnPU (0.828), despite Separator being competitive or superior at lower ρ values (e.g., ρ = 0.3: 0.862 accuracy). The same pattern holds at nl = 500 and ρ = 0.7 (Table III: Separator 0.760 vs. Weighter 0.833) and nl = 1,000 and ρ = 0.7 (Table IV: Separator 0.743 vs. Weighter 0.843). The learning curves in Figure 2 (bottom row, ρ = 0.7) show Separator struggling to improve beyond ~0.72 accuracy even after 300 epochs, while Weighter continues to climb gradually. On CIFAR-10, the Separator is more robust at high ρ (Table VII: 0.833 at ρ = 0.7 vs. Weighter 0.816), suggesting the failure mode is dataset-dependent—but the structural vulnerability of the threshold-based reward remains regardless.

Mitigation status. The paper identifies the problem and explains its mechanism but proposes no solution. The only mitigation is to use the Weighter variant when ρ is expected to be high, since continuous weights are less sensitive to the reward threshold's behavior. The paper does not experiment with alternative reward designs (per-example adaptive thresholds, reward shaping to discount the threshold's influence on boundary examples, or a different reward function entirely for the Separator). The limitation is therefore acknowledged but unresolved, and practitioners working with datasets where positives may dominate the unlabeled pool must be aware that the Separator variant may underperform estimation-based methods in this regime.

Difficulty Estimation Cost Is Unaccounted for in the Headline Comparisons

The assumption or constraint. The paper's compute-optimal framework for test-time compute scaling is NOT directly claimed here. However, the policyPU framework incurs a training-time cost that is not included in any efficiency comparison: the policy network must be trained jointly with the classifier, requiring forward and backward passes through both networks for every mini-batch, plus the overhead of REINFORCE gradient computation and periodic target network synchronization. The paper does not report wall-clock training time, epoch time, or FLOPs comparisons between policyPU and the baseline methods.

The consequence. The reported performance improvements (e.g., Weighter's 0.916 accuracy vs. KM2+nnPU's 0.867 on MNIST at nl = 300, ρ = 0.3) come at an unknown computational cost premium. A practitioner deciding between policyPU and a simpler baseline needs to know whether the 4.9 percentage point accuracy gain justifies the additional training time. On CIFAR-10, TIcE+nnPU fails to 0.592 accuracy at nl = 1,000 and ρ = 0.5 (Table VII), and a practitioner might reasonably prefer to run TIcE+nnPU five times with different random seeds and ensemble the results (potentially matching policyPU's performance at similar or lower total cost) rather than implement the RL-based training framework. Without training cost data, the paper's accuracy improvements cannot be translated into an efficiency metric, which is essential for practical decision-making.

The interactive training also likely converges more slowly than one-shot methods. While estimation-based methods like TIcE+nnPU separate prior estimation from classifier training (each can be optimized independently), policyPU requires the policy and classifier to co-evolve, and the "dip and recovery" pattern in Figure 5 shows the policy spends early epochs making worse-than-initial decisions before improving. On UserTargeting, the paper runs 1,000 epochs vs. 300 on MNIST/CIFAR-10, acknowledging slower convergence. This longer training compounds the per-epoch cost premium.

What evidence exists in the paper. The paper provides no direct evidence on training cost. There are no wall-clock time measurements, no FLOPs counts, no comparisons of convergence speed in terms of iterations-to-target-accuracy. The architecture descriptions (Section V-C) indicate the policy network adds parameters (a 5-layer CNN on top of a 6-layer CNN classifier), and the Algorithm 1 pseudocode shows both networks are updated every mini-batch, but the cost is not quantified. The pre-training phase (5 epochs of biased PU for the classifier, plus policy pre-training) is also unaccounted for in the "300 epochs" figure since it happens before the interactive training begins.

Mitigation status. Not addressed. The paper's focus is purely on final classification performance, and training cost is not discussed as a factor. For researchers, this is acceptable—the contribution is methodological. For practitioners deploying PU learning in production or at scale, the absence of cost data is a significant gap. Future work could profile policyPU against baselines in terms of GPU-hours to reach a target accuracy, providing the efficiency data needed for adoption decisions.

Generalization Is Demonstrated on Only Three Datasets, Two of Which Are Clean Image Benchmarks

The assumption or constraint. All experimental validation is conducted on three datasets: MNIST, CIFAR-10, and UserTargeting. Two of these are image classification benchmarks with clean, structured features, binarized in a manner the paper inherits from Kiryo et al. [24]. The third is a proprietary e-commerce dataset with no publicly available description beyond what the paper provides. The paper makes no claims about broader domain applicability beyond these three datasets, and the diversity of tasks tested is limited to: (1) handwritten digit recognition (MNIST), (2) object recognition (CIFAR-10), and (3) user response prediction (UserTargeting).

The consequence. Several aspects of policyPU's performance may not transfer to other PU learning domains:

Text classification and NLP tasks: Text data has discrete, sparse feature representations (bag-of-words, TF-IDF) or requires embedding layers and sequence models, which have very different optimization landscapes from CNNs on continuous pixel data. The policy network's ability to learn useful per-example weights depends on the feature representation's structure, and nothing in the paper demonstrates this capability on discrete or sequential features.
Highly imbalanced datasets: The paper tests ρ ∈ {0.3, 0.5, 0.7}, meaning positives are always a substantial fraction (30–70%) of the unlabeled data. Many real-world PU problems have much lower positive ratios—for instance, click-through prediction where positives are ~1% of unlabeled users, or disease diagnosis where incidence is a fraction of a percent. At very low ρ, the reward function's threshold would be computed from a tiny labeled positive set (if nl is also small), making the threshold unstable and the reward signal noisy. The Separator's degradation at ρ = 0.7 hints that the method is sensitive to the positive ratio; behavior at ρ = 0.01 is entirely unknown.
High-dimensional or structured output spaces: The paper addresses binary classification only. PU learning for multi-class, multi-label, or structured prediction is not tested, and the Weighter's mixture-model loss (Equation 8) does not trivially extend beyond binary classification.
Datasets where SCAR is violated: The paper inherits the SCAR assumption (Section II) but does not test on datasets where labeled positives are selected with bias (e.g., positives with certain features are more likely to be labeled). The policy might learn to exploit SCAR violations in ways that hurt generalization, or the coherence reward might break down if labeled positives are not representative of all positives.

What evidence exists in the paper. The paper provides strong evidence that policyPU works on the three tested datasets (Tables II–VIII), but no evidence about its behavior outside this domain. The diversity among the three datasets is limited: MNIST and CIFAR-10 use identical CNN architectures, identical preprocessing, and are both computer vision tasks; UserTargeting adds a tabular domain but with only 153 features and a relatively balanced class structure (25% labeled positives among all data). The paper does not discuss how the method's design choices (continuous actions, coherence reward, dynamic threshold) would scale or adapt to different data modalities.

Mitigation status. The paper does not address generalization to other domains and makes no claims about broader applicability beyond what the experiments demonstrate. The limitation is primarily one of scope: the method is validated for binary image classification and one tabular prediction task under moderate positive ratios, and practitioners in other domains (NLP, extreme class imbalance, structured prediction) would need to conduct their own validation. The paper's claim of "consistent improvements" is true within the tested scope but should not be interpreted as evidence of universality.

The Method Adds Two Nontrivial Hyperparameters (Policy Update Frequency and Pre-Training Duration) with No Sensitivity Analysis

The assumption or constraint. The interactive training in Algorithm 1 depends on two hyperparameters that have no counterpart in the estimation-based baselines: k, the frequency (in epochs) at which the target policy network is synchronized with the current policy ( $\Theta' \leftarrow \Theta$ ), and the pre-training duration for both the classifier (biased PU pre-training) and the policy network (imitation of the pre-trained classifier). The paper fixes k = 3 and the pre-training duration at 5 epochs for image data without ablation or justification. The weight decay values for policy networks differ substantially across datasets and variants (MNIST: 2.0 for Weighter, 0.5 for Separator; CIFAR-10: 0.005 for Weighter, 1.0 for Separator; UserTargeting: 1e-4 for both), suggesting dataset-specific hyperparameter tuning whose sensitivity is unknown.

The consequence. A practitioner applying policyPU to a new dataset must either inherit these hyperparameter values (hoping they transfer) or conduct expensive hyperparameter searches. The sensitivity is likely high:

Target policy update frequency (k): If k = 1 (update every epoch), the policy chases a rapidly shifting action distribution, producing high-variance REINFORCE gradients that may prevent convergence. If k is large (e.g., k = 10), the policy learns from stale rewards generated under an outdated action distribution, slowing adaptation. The optimal k depends on how quickly the classifier's decision boundary evolves, which in turn depends on dataset size, difficulty, and learning rate—none of which transfer automatically across domains.
Pre-training duration: Too little pre-training and the interactive loop starts with a near-random classifier providing meaningless rewards, creating a destructive feedback loop. Too much pre-training and the classifier converges to the biased PU solution, which the policy then learns to imitate, making it difficult for the interactive training to escape the bias. The 5-epoch default was likely chosen because it works empirically on MNIST and CIFAR-10, but the paper provides no evidence that this is robust.
Policy network weight decay: The large variation (0.005 to 2.0, a 400× range) suggests this hyperparameter is actively tuned per-dataset, yet the paper provides no tuning protocol. A practitioner would need to run multiple experiments to find a working value, negating some of the simplicity advantage policyPU claims over estimation-based methods.

What evidence exists in the paper. None. The paper reports final results after 300 (or 1,000) epochs with the stated hyperparameters but does not vary k, pre-training duration, or policy weight decay to assess sensitivity. The target network technique is described as being used "to reduce high variance of the returned reward" (Section III-E), but there is no comparison to training without a target network (k = 1) or to other variance reduction techniques. The paper's only ablation of a training hyperparameter is the implicit comparison of 4-layer vs. 6-layer policy networks on UserTargeting (Table VIII), which shows small differences but does not address the more fundamental sensitivity to k or pre-training.

Mitigation status. Not addressed. The authors do not discuss hyperparameter sensitivity, tuning protocols, or transferability. The paper would be strengthened by a sensitivity analysis showing that performance is stable across a reasonable range of k (e.g., 2–5) and pre-training durations (e.g., 3–10 epochs), which would build confidence that the reported results aren't artifacts of cherry-picked hyperparameters.

The UserTargeting Evaluation Methodology Is Ambiguous Because Ground-Truth Negatives Are Unavailable

The assumption or constraint. The UserTargeting dataset is described as containing "4,758 labeled and 14,274 unlabeled users" in both train and test splits (Section V-A). Unlike MNIST and CIFAR-10—where the binarization creates known ground-truth negatives—UserTargeting appears to be a genuine PU dataset where only some positives are labeled and the true class of unlabeled users is unknown. The paper does not explain how classification metrics (accuracy, ROC AUC, PR AUC) are computed on test data when the ground-truth negative labels don't exist.

The consequence. Standard PU evaluation requires either: (a) access to ground-truth labels for the test set (making evaluation identical to PN evaluation), or (b) using PU-specific metrics that don't require known negatives, such as the probability of correct classification given a positive is labeled, or evaluation on a separate fully labeled dataset. The paper's use of accuracy, ROC AUC, and PR AUC on UserTargeting implies one of three possibilities:

Possibility 1: The dataset actually has ground-truth negatives (e.g., users who definitely didn't respond) but the paper describes it imprecisely. If so, the evaluation is valid but the dataset description is misleading.
Possibility 2: All unlabeled test users are treated as negatives for evaluation purposes. This would make the reported metrics systematically biased—a classifier that correctly identifies hidden positives in the unlabeled set would be penalized for "misclassifying" them, and the metric would favor classifiers that reproduce the naive biased-PU assumption.
Possibility 3: The evaluation uses some external source of negative labels not described in the paper. Without documentation, the results are not reproducible.

What evidence exists in the paper. The paper provides no explanation of the UserTargeting evaluation methodology. The dataset description states: "The labeled examples in the UserTargeting dataset are those users who responded positively to certain products on an e-commerce platform. They are used to identify potential users among all other users of this e-commerce platform." This describes a standard PU setup where unlabeled users may or may not be interested. The test set is described only as: "This dataset is split 50-50 to train and test. Both of them have 4,758 labeled and 14,274 unlabeled users." There is no mention of how negatives are identified for computing accuracy, ROC AUC, or PR AUC on the test set. Table VIII reports concrete numbers (e.g., Separator: 0.937 accuracy, 0.974 ROC AUC) as if they are well-defined, but the computation is opaque.

Mitigation status. Not addressed. This is the most severe methodological gap in the paper because it potentially invalidates the UserTargeting results entirely. If the evaluation treats unlabeled test users as negatives, the reported metrics are measuring a mix of true classification performance and conformity to the biased-PU assumption, making them incomparable to the MNIST/CIFAR-10 results where ground-truth negatives are available. The paper should either clarify the evaluation protocol or acknowledge the ambiguity as a limitation. Without this clarification, practitioners evaluating policyPU for real-world PU problems—where ground-truth negatives are unavailable by definition—cannot trust that the reported UserTargeting performance translates to their scenario.

No Comparison Against Iterative or Ensemble Baselines That Also Use Feedback Loops

The assumption or constraint. The paper frames policyPU's key advantage as the interactive, dynamic training between the policy and classifier, contrasting it with the "unidirectional" pipeline of estimation-based methods. However, the baseline selection includes only methods that do estimation once and then train a classifier: TIcE+nnPU, KM2+nnPU, RankPruning, PMPU, and biased PU. None of the baselines use iterative refinement, bootstrapping, co-training, or ensemble mechanisms that also exploit feedback between label assumptions and classification.

The consequence. The experimental comparison does not isolate the value of interactivity from the value of using a learned neural network to make per-example label decisions. Several alternative approaches could plausibly close the gap with policyPU without requiring reinforcement learning:

Iterative re-estimation: Run TIcE (or KM2) to estimate the class prior, train a classifier with nnPU, use that classifier's predictions to refine the prior estimate, and repeat. This creates a feedback loop using off-the-shelf components without RL.
Self-training or bootstrapping: Train an initial classifier with biased PU, use it to predict labels for unlabeled examples, retrain on the augmented dataset, and iterate. This is a standard semi-supervised technique that naturally applies to PU learning.
Ensemble methods: Train multiple classifiers with different class prior estimates (bootstrap samples of the PU data) and aggregate their predictions. The diversity across priors could provide the same regularizing benefit the paper attributes to policy errors on boundary examples.

Without comparisons against these baselines, it is unclear whether policyPU's advantage comes from the specific RL-based interactive mechanism, from the per-example neural network label assigner (which would also benefit one-shot methods), or simply from using a more expressive label-assignment model than a single scalar class prior.

What evidence exists in the paper. None of these iterative or ensemble baselines appear in the experiments. The comparison is exclusively against pipeline methods that estimate a prior once and train a classifier with it. PMPU [32] is the closest to an iterative method—it alternates between estimating unlabeled labels and retraining the classifier—but it relies on a specific margin assumption that fails on UserTargeting (0.631 accuracy, Table VIII), making it a weak representative of iterative approaches in general. The paper's central claim that policyPU's interactive nature is the key insight would be substantially stronger if it could show that policyPU outperforms simpler iterative methods that also close the feedback loop but without RL.

Mitigation status. Not addressed. The paper does not discuss iterative or ensemble alternatives, nor does it argue why RL is necessary rather than one possible implementation of the interactive paradigm. The conceptual contribution (Section 4, Innovation 1)—that PU learning should be an interactive decision problem rather than a one-shot estimation problem—is broader than the RL implementation, and the experiments would benefit from evidence that the RL implementation specifically adds value over other ways to realize interactive training. This gap in the baseline set limits the strength of the claim that policyPU's approach is uniquely effective rather than simply being the first interactive method tested against non-interactive baselines.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing of PU learning rather than an incremental improvement to existing estimation techniques. Prior to policyPU, the field operated under a dominant paradigm inherited from Elkan and Noto [8]: PU learning was fundamentally an estimation problem—estimate the class prior c, plug it into a corrected loss function or calibrated classifier, and the problem reduces to supervised learning. This paradigm produced a clear research agenda (develop better class prior estimators, derive better risk correctors) but also a structural limitation: estimation and classification were decoupled, with no mechanism for the classifier's needs to influence the estimation. The paper provides evidence that this decoupling is the binding constraint on performance, not the accuracy of any particular estimator. On CIFAR-10 with 1,000 labeled examples and ρ = 0.5, TIcE+nnPU—a state-of-the-art estimation pipeline—achieves only 0.592 accuracy, while KM2+nnPU—a different estimator plugged into the same classifier training method—achieves 0.745. The 15-point gap comes entirely from estimation quality, and the classifier has no way to compensate. PolicyPU Weighter achieves 0.853 by bypassing global estimation entirely and instead learning per-example weights optimized for classification utility through interactive feedback.

The magnitude of this shift is a reframing with practical consequences, not a paradigm overthrow. The paper does not claim that class prior estimation is irrelevant or that SCAR-based theory is wrong—the Weighter variant directly inherits the mixture-model formulation from Elkan and Noto. Rather, it demonstrates that how you arrive at per-example label assumptions matters as much as what those assumptions are, and that optimizing them jointly with the classifier through a feedback loop recovers substantial performance that fixed pipelines leave on the table. This is analogous to the shift in NLP from pipeline approaches (tokenize → POS tag → parse → classify) to end-to-end neural models: the individual steps may be theoretically justified, but jointly optimizing them for the final objective produces better results. PolicyPU makes the same argument for PU learning: stop estimating the class prior as an isolated goal and start learning label assignments as part of classifier training.

The paper also reconciles contradictory intuitions about what makes PU learning hard. Prior work split roughly into two camps: one focused on the statistical challenge of estimating c under the SCAR assumption (TIcE, KM2, the unbiased risk estimator family), implicitly assuming that accurate estimation is sufficient; the other focused on cleaning the dataset by identifying mislabeled examples (RankPruning, PMPU), implicitly assuming that label noise is the primary obstacle. PolicyPU's results suggest both camps were partially right and partially wrong. The difficulty is neither pure estimation nor pure noise removal—it is decision-making under uncertainty where the quality metric is downstream classification performance, not statistical fidelity to the data distribution. The policyPU framework subsumes both perspectives: the policy learns to make decisions that implicitly encode an estimate of which examples are positive (label cleaning) while optimizing for classification utility rather than statistical accuracy (going beyond estimation). This explains why RankPruning can work well on UserTargeting (0.880 accuracy, Table VIII) while failing on CIFAR-10 at higher label budgets (0.712 at nl = 1,000, ρ = 0.5, Table VII)—its pruning decisions optimize a proxy (confidence ranking) rather than downstream classifier performance.

Several research directions become more attractive in light of this work:

Learned per-example weighting for other weakly-supervised problems. The core mechanism—a policy network learns how to weight training examples by receiving rewards from the downstream classifier—is not PU-specific. It applies to any setting where training labels are noisy, incomplete, or ambiguous and a clean validation signal exists. Learning-to-reweight for noisy labels, class-imbalanced learning, or domain adaptation could adopt the same interactive RL framework with modified reward functions.
Richer interaction protocols between label-assignment and classification. The paper uses a simple per-mini-batch reward loop, but the conceptual reframing opens a design space: bidirectional training with different update schedules (asynchronous, multi-step), multi-agent formulations where multiple policies compete or cooperate, or meta-learning the policy's update rule itself. The specific REINFORCE implementation is one point in this space; exploration of the space is now motivated.
Understanding and exploiting the regularizing effect of learned label noise. The paper's observation that policyPU sometimes exceeds optimal PN performance (Section V-E)—attributed to the policy's errors on boundary examples acting as data-dependent regularization—opens a line of inquiry into constructive label noise. Rather than treating label noise as a bug to be removed (RankPruning's approach), could we design systems that deliberately introduce classifier-adaptive label noise to improve generalization? The policyPU framework provides a natural testbed for this hypothesis.

Conversely, some directions become less attractive. The paper's results suggest that marginal improvements to class prior estimation—a better kernel for KM2, a more robust tree induction for TIcE—are unlikely to close the gap with jointly optimized approaches. The estimation pipeline's performance ceiling appears to be bounded not by estimator quality but by the structural limitation of decoupling estimation from classification. Research effort is better directed at developing interactive, reward-driven methods than at incrementally refining one-shot estimators for the same pipeline.

Follow-Up Research This Work Enables

1. Direct measurement of whether policyPU learns the class prior as a byproduct of interactive training. The paper never measures whether the policy's implicit class prior estimate (the fraction of unlabeled examples assigned as positive, or the mean continuous weight) converges to the true ρ during training. This is a critical missing diagnostic: if the policy does learn an accurate class prior, then the interactive training is effectively performing prior estimation as a byproduct of optimizing classification performance—a more robust estimator than TIcE or KM2 because it is self-correcting. If the policy does not learn an accurate prior, then policyPU's advantage comes entirely from per-example weighting that outperforms global prior knowledge, a stronger and more surprising result. A follow-up experiment would track the policy's mean assigned-positive rate per epoch on MNIST and CIFAR-10 (where ρ is known) and plot it against the true ρ, for both Weighter and Separator, across different ρ values. The result would distinguish between "policyPU is a better prior estimator" and "policyPU makes prior estimation unnecessary."

2. Stress-testing the Separator's failure mode at extreme positive ratios (ρ → 1.0) with alternative reward designs. The paper identifies the Separator's degradation at ρ = 0.7 as a consequence of the threshold-based reward elevating above boundary-case positives, creating perverse incentives. But the paper stops at diagnosis. A strong follow-up would systematically vary ρ from 0.1 to 0.9 on MNIST at fixed nl = 300 and compare the current dynamic threshold (Equation 5) against: (a) a fixed threshold at 0.5, (b) a threshold computed from labeled positives only (without U'), (c) per-example rewards that use ŷ directly with no threshold at all (R = ŷ for all unlabeled examples), and (d) a reward that is the signed distance from the threshold (continuous, zero-crossing) rather than a binary split. The predicted outcome is that removing the threshold or softening it should improve Separator performance at high ρ, at the cost of reduced performance at low ρ—quantifying this tradeoff would establish whether the threshold is a necessary evil or a correctable design flaw.

3. Replacing REINFORCE with an actor-critic method to reduce gradient variance and improve convergence speed. The paper uses vanilla REINFORCE with a target network (updated every k = 3 epochs) as the only variance-reduction technique, and the learning curves (Figure 5) show the policy's assignment accuracy dips early in training before recovering—suggesting high-variance gradients cause exploratory mistakes. A natural extension is to train a critic network that estimates the value function V(x) = E[R(x,a)] and use the advantage A(x,a) = R(x,a) − V(x) in place of the raw reward in the policy gradient. The critic could be trained on the same coherence rewards with simple MSE. The predicted benefit is faster convergence (the policy avoids the "dip and recovery" phase because advantage-weighted gradients have lower variance), enabling interactive training to be practical on larger datasets where 300+ epochs are costly. A comparison of REINFORCE vs. actor-critic on MNIST at nl = 300 and ρ = 0.5 would directly test whether gradient variance is the bottleneck in policy learning speed.

4. Evaluating policyPU against iterative self-training baselines that also close the feedback loop without reinforcement learning. The paper's key claim is that interactive, feedback-driven training outperforms one-shot estimation pipelines. But interaction does not require RL. A simple baseline—train a biased PU classifier, use it to predict labels for unlabeled examples, retrain on the augmented labeled set, and iterate for a fixed number of rounds—implements a feedback loop using only standard supervised training. Similarly, TIcE+nnPU could be run iteratively: estimate c, train a classifier, use the classifier's predictions to refine the estimate of c (perhaps by re-weighting examples in TIcE's decision tree), and repeat. Comparing policyPU against these baselines on CIFAR-10 at nl = 300 and ρ = 0.5 would isolate the value of RL-based interaction from the value of any interaction. If self-training matches policyPU, the paper's contribution shifts from "RL is necessary" to "interaction is necessary, and RL is one effective implementation." If self-training underperforms, the paper's specific RL design (per-example weights, coherence reward) is validated as essential.

5. Testing whether the policy's per-example weights can be distilled into a single inference-time classifier that does not require the policy network at deployment. PolicyPU trains two networks (policy π_Θ and classifier F_Φ) interactively, but at inference time, only the classifier is needed—the policy's role is complete once training finishes. This means the policy network's learned knowledge (which unlabeled examples should be weighted how) is implicitly encoded in the classifier's parameters through the interactive training. A follow-up could attempt to train a classifier from scratch on the original PU data without the policy network, using a loss function derived from the converged policy's behavior—for instance, using the policy's per-example weights as fixed soft labels, or distilling the policy's assignment decisions into a set of rules. If this distilled classifier matches the interactively trained one, it would demonstrate that policyPU's value is in generating high-quality training targets rather than in the interactive process per se, enabling cheaper deployment. A concrete experiment: take the converged Weighter policy and classifier from MNIST (nl = 300, ρ = 0.3), use the policy to label all unlabeled training examples with continuous weights, and train a new classifier (same architecture) on this weighted dataset without the policy network. Compare the test accuracy of this distilled classifier to the original interactively-trained one.

6. Extending policyPU to multi-class and structured-output PU learning to test the generality of the coherence reward. The paper addresses binary classification exclusively, and the Weighter's loss function (Equation 8) relies on the binary mixture-model formulation w·log(ŷ) + (1−w)·log(1−ŷ). Multi-class PU learning (where positives may belong to multiple classes and negatives are none-of-the-above) requires a different reward design because "coherence" between policy assignment and classifier prediction is no longer a simple ŷ vs. 1−ŷ decision. A natural extension would define the coherence reward as the classifier's predicted probability of the class the policy assigned, with the threshold computed per-class using labeled positives from each class. On a multi-class PU variant of CIFAR-10 (e.g., animals are negatives, vehicles are positives but with different vehicle types as separate classes), comparing policyPU to a multi-class extension of nnPU would test whether the interactive paradigm's advantage persists when the action space expands and the coherence reward becomes sparser.

Practical Applications and Downstream Use Cases

1. E-commerce user targeting and recommendation systems where negative feedback is absent. The UserTargeting dataset directly represents a common production scenario: an e-commerce platform has users who clicked or purchased (labeled positives) and a vast pool of users with no explicit feedback (unlabeled). Building a classifier to identify likely-responding users among the unlabeled pool is a core business problem for targeted advertising, email campaigns, and recommendation. policyPU Separator achieves 0.937 accuracy and 0.974 ROC AUC on this dataset (Table VIII), outperforming the best estimation baseline (KM2+nnPU: 0.896 accuracy) by 4.1 percentage points. In a production system serving millions of users, a 4-point accuracy improvement in targeting translates directly to increased click-through rates or reduced wasted impressions, with the practical benefit that policyPU requires no changes to the data collection pipeline—the same labeled positives and unlabeled users are used, just with a different training algorithm.

2. Medical diagnosis from electronic health records where only diagnosed cases are labeled. The paper's motivating example (Section I) is directly actionable: in diagnostic systems, confirmed diagnoses provide labeled positives, but undiagnosed patients cannot be assumed disease-free. The policyPU framework can be deployed on historical EHR data where some patients have confirmed diagnosis codes (positive) and the rest have no such codes (unlabeled). The practical advantage over estimation-based methods is robustness to the unknown true disease prevalence: unlike TIcE or KM2 which require accurate class prior estimation (problematic when prevalence varies across patient subgroups), policyPU learns per-patient weights that adapt to local feature space structure. The MNIST results at low label budgets (nl = 300, Table II) are particularly relevant for rare disease settings where few confirmed cases exist: Weighter achieves 0.916 accuracy with only 300 labeled positives, approaching optimal PN (0.919), while the best estimation baseline (KM2+nnPU) reaches only 0.867. The 4.9-point gap suggests that for rare diseases with limited confirmed cases, the interactive paradigm extracts substantially more diagnostic signal from unlabeled patient records than traditional prior-estimation approaches.

3. Data annotation and active learning pipelines where labeling negatives is expensive. In many annotation workflows, confirming a positive example requires one judgment ("yes, this image contains a cat"), but confirming a negative requires exhaustive verification ("no, after checking all possible objects, this image does not contain any of the target classes"). policyPU can be integrated into an annotation pipeline where human annotators label only positives (cheap), and the system learns from a large pool of unlabeled images (free). The key practical benefit is that policyPU's performance approaches optimal PN (fully supervised on both classes) with only positive labels—on MNIST at nl = 1,000 and ρ = 0.3, Weighter achieves 0.948 accuracy vs. optimal PN's 0.960 (Table IV), a gap of only 1.2 percentage points while saving the cost of labeling thousands of negative examples. For an annotation team labeling 10,000 images, switching from full PN labeling (label every image as positive or negative) to PU labeling with policyPU (label only positives, leave the rest unlabeled) could reduce labeling effort by 50–70% (depending on the natural positive ratio) while sacrificing only ~1 point of accuracy, based on these MNIST results.

4. Self-improving content moderation systems for platforms where violations are sparsely reported. Content platforms (social media, marketplaces, forums) rely on user reports to flag policy-violating content. Reported-and-confirmed violations are labeled positives, but unreported content may be benign (true negative) or undetected violations (false negative, unlabeled positive). Training a classifier to detect violations from reported positives and unreported content is a PU learning problem. policyPU is directly applicable: the policy network learns to identify which unreported content items are likely hidden violations based on the classifier's evolving ability to discriminate, while the classifier improves from the policy's discoveries. The practical advantage over biased PU (treating all unreported content as benign, which would systematically miss violations that look similar to reported cases) is substantiated by the MNIST and CIFAR-10 results where biased PU consistently underperforms—at nl = 300 and ρ = 0.5 on MNIST, biased PU achieves 0.535 accuracy vs. policyPU Weighter's 0.880 (Table II), a catastrophic gap that represents the difference between a useful moderation system and one no better than random for detecting unreported violations.

When to Prefer This Method

The paper positions policyPU explicitly against the "unidirectional" estimation pipeline, arguing that interactive, reward-driven training recovers performance that fixed pipelines cannot. The conditions under which policyPU is preferable follow directly from the experimental results and identified limitations:

Prefer policyPU (specifically the Weighter variant) when:

You have a moderate number of labeled positives (hundreds to low thousands, as tested with nl = 300–1,000) and a pool of unlabeled data where the positive ratio is not extreme (ρ = 0.3–0.5 performed best). The paper shows Weighter consistently outperforms all estimation baselines in this regime, with gains up to 5–10 percentage points in accuracy.
The SCAR assumption may be approximately but not perfectly satisfied, making class prior estimation unreliable. PolicyPU learns per-example weights optimized for classification utility rather than statistical fidelity, and the CIFAR-10 results (where TIcE+nnPU collapses to 0.592 accuracy at nl = 1,000 and ρ = 0.5, Table VII) demonstrate that estimation pipelines can fail catastrophically when prior estimation is poor, while policyPU remains robust (Weighter: 0.853 in the same configuration).
You can afford the additional training complexity (implementing REINFORCE, tuning the target network update frequency and pre-training duration) in exchange for substantial accuracy improvements. The paper does not provide cost data, so the practical decision requires a local cost-benefit analysis.
The dataset has sufficient feature structure that a neural network policy can learn meaningful per-example distinctions—the policy is a neural network, and its effectiveness depends on the feature representation supporting generalization across examples.

Prefer estimation-based methods (TIcE+nnPU, KM2+nnPU) when:

The positive ratio in unlabeled data is expected to be very high (ρ ≫ 0.7) and you plan to use hard label assignments. The Separator variant degrades in this regime due to the threshold-based reward design (Tables II–IV: Separator drops to 0.722–0.760 accuracy at ρ = 0.7 on MNIST), and while Weighter is more robust, the paper does not test Weighter at ρ > 0.7.
Implementation simplicity and training speed are primary concerns. Estimation pipelines require only running an off-the-shelf prior estimator and then a standard classifier training loop—no RL, no target networks, no policy gradient. For rapid prototyping or deployment on resource-constrained systems, this simplicity may outweigh policyPU's accuracy gains.
You have very few labeled positives (nl ≪ 100), where the dynamic threshold in the coherence reward (Equation 5) may become unstable because thresh_min is computed from a tiny set of labeled positives. The paper never tests below nl = 300, and performance at very low label counts is unknown.

Prefer biased PU (treat unlabeled as negative) only when:

External knowledge confirms that unlabeled examples are overwhelmingly negative (ρ ≈ 0), in which case the naive assumption is approximately correct and more complex methods add overhead without benefit. The UserTargeting results (Table VIII) show biased PU is surprisingly competitive (0.872 accuracy vs. policyPU's 0.937), suggesting the domain's natural class balance makes the naive assumption reasonable—but policyPU still provides a 6.5-point gain, so "prefer" here means "acceptable if implementation cost is prohibitive," not "equally good."