Hard negative mining

Object detection

In classical detectors, hard negative mining is an iterative, cache-based procedure layered on top of an SVM or other classifier: the model is trained on the positives and a small set of negatives, the trained model is run over background data to collect new false positives, easy examples are discarded, and the model is retrained, repeating a few times. A negative is treated as hard when its margin score indicates misclassification or a within-margin position.

Online hard example mining (OHEM) adapts this idea to the training of deep region-based detectors. Rather than alternating offline mining and retraining, OHEM forms each SGD mini-batch by scoring all candidate regions of interest with a read-only forward pass, and then back-propagating only through the highest-loss regions. An efficient implementation keeps two copies of the region network: one read-only copy that scores all regions and one trainable copy that processes only the selected hard examples. Hard negative mining has likewise been applied to convolutional face detectors by harvesting false positives from a first-round Faster R-CNN face detector and retraining on them, labelling a detected region a hard negative when its maximum intersection-over-union with any ground-truth face falls below a certain threshold.

Deep metric learning

In deep metric learning, models are trained with pairwise or triplet losses so that examples of the same class map to nearby points and examples of different classes map to distant points. Because most randomly sampled triplets quickly satisfy the loss margin and stop contributing gradient, the selection of hard pairs or triplets is widely regarded as essential. The triplet loss, in fact, penalizes triplets in which the anchor–positive distance is not smaller than the anchor–negative distance by at least a margin ${\displaystyle \alpha }$.

The FaceNet system introduced semi-hard negative mining, in which the chosen negative is farther from the anchor than the matching positive yet still falls within the margin, after finding that always selecting the hardest negatives could drive the embedding toward a collapsed (degenerate) solution; FaceNet also computed hard positives and negatives online from within each training batch rather than searching the whole dataset.

An early example of in-batch top-K selection appears in unsupervised video representation learning, where the negatives incurring the highest triplet loss within a mini-batch are selected. A widely cited variant is batch-hard mining, where a batch is built by sampling a fixed number of classes and a fixed number of examples per class, and for each anchor only the hardest positive and hardest negative within that batch are used; because these are the hardest within a small random subset rather than the whole dataset, the authors argued they act as "moderate" triplets and avoid the cost and instability of offline mining.

Several methods reduce the cost of mining or change what is mined:

• The multi-class N-pair loss: a single update contrasts the anchor with many negative classes at once; for datasets with very many classes its author additionally proposed a greedy hard negative class mining scheme that selects whole negative classes rather than individual samples.
• Lifted structured embedding mines hard negatives relative to both endpoints of a positive pair and, observing that selecting only the single hardest negative can drive the network to a poor local optimum, optimizes a smoothed log-sum-exp objective aggregating all negatives within the margin.
• Smart mining frames hard negative mining as an approximate nearest neighbour search over the embeddings, building a neighbour graph once per epoch and selecting negatives outside a per-anchor exclusion radius.
• Hard-aware deeply cascaded embedding (HDC) selects hard examples by ranking per-sample losses within each mini-batch and routes samples of different difficulty to cascaded sub-networks of increasing complexity, motivated by the observation that hardness is relative to model capacity.
• Stochastic class-based mining reduces the cost of searching for hard negatives by first identifying a small set of neighbouring classes and searching for hard instances only within them, while stochastically varying the candidate pool across iterations.

Information retrieval and recommendation

In recommender systems and neural information retrieval, training pairs each observed (relevant) item against non-observed items as implicit negatives. Bayesian Personalized Ranking constructs training triples in which an observed item is treated as a positive and a non-observed item as an implicit negative, and trains a pairwise objective by drawing such triples uniformly at random; it does not use hard or dynamic negative selection, and reviews describe it as the uniform-sampling baseline that later hard-negative methods improved upon. In neural dense retrieval, a dual encoder is trained so that relevant query-document pairs have higher similarity than irrelevant ones, and the choice of negatives is decisive.

Several strategies are in common use:

• In-batch negatives plus a lexical hard negative. Dense passage retrieval (DPR) trains using the other questions' passages in the same mini-batch as negatives, together with a single hard negative selected by BM25, a passage that scores highly for the question lexically but does not contain the answer.
• Global, dynamically refreshed hard negatives. Approximate Nearest Neighbor Negative Contrastive Estimation (ANCE) mines hard negatives from the whole corpus using an approximate nearest neighbour index over the model's own embeddings that is rebuilt periodically from a recent checkpoint, so the mined negatives stay aligned with the evolving model.
• Denoised hard negatives. RocketQA noted that treating top-ranked passages as negatives can introduce relevant but unlabelled passages as "false negatives," and used a stronger cross-encoder to score mined candidates, keeping only those confidently predicted to be negatives.
• Static versus dynamic hard negatives. Zhan and colleagues drew an explicit distinction between static hard negatives, pre-retrieved once and held fixed during training, and dynamic hard negatives, recomputed from the current model, and analyzed the two as optimizing different objectives, arguing that hard negative sampling emphasizes top-ranking performance.

Contrastive self-supervised learning

Contrastive self-supervised learning methods learn representations by pulling together different views of the same instance (positives) and pushing apart unrelated ones (negatives). The widely used InfoNCE objective, introduced in contrastive predictive coding, scores one positive against ${\displaystyle N-1}$ negatives drawn from a proposal distribution:

${\displaystyle {\mathcal {L}}_{N}=-\,\mathbb {E} \!\left[\,\log {\frac {f_{k}(x_{t+k},c_{t})}{\sum _{x_{j}\in X}f_{k}(x_{j},c_{t})}}\,\right].}$

In its original form, contrastive predictive coding draws its negatives at random and recommends using more negatives because the objective bounds a mutual information quantity that tightens as ${\displaystyle N}$ grows. Several influential methods similarly increase the supply of negatives rather than their difficulty: SimCLR treats all other augmented examples in a large training batch as negatives and argues that its temperature-scaled cross-entropy loss implicitly weights negatives by hardness, whereas margin- and triplet-style losses require explicit semi-hard mining to remain competitive; and Momentum Contrast (MoCo) maintains negatives in a queue decoupled from the batch size, encoded by a slowly updated momentum encoder to keep them consistent.

Explicit hard-negative methods have nonetheless been adapted to the unsupervised setting.

• Importance weighting. A tunable sampling distribution that up-weights negatives the current embedding maps close to the anchor while approximately discarding likely false negatives, implemented as a reweighting of the contrastive loss; a hardness parameter interpolates between uniform sampling and an adversarial worst-case distribution.
• Synthetic embeddings. Rather than selecting difficult examples, MoCHi synthesizes them in the embedding space, forming new negatives as normalized convex combinations of the hardest existing negatives and mixing the query with hard negatives to obtain still-harder examples, after observing that most stored memory negatives contribute little to the loss as training progresses.
• Adversarial perturbations. The CLAE method applies adversarial perturbations computed jointly across the batch, which its authors characterize as performing hard negative mining without labels.

Vision-language models

In CLIP-style vision-language models, hard negatives are used to address the "bag of words" behavior of such models, ignoring word order and object-attribute binding. Methods in this area differ in whether they select, construct, or reweight negatives:

• Constructing hard negative captions. NegCLIP generates negative captions by swapping nouns, adjectives and verb phrases within a correct caption, and adds nearest-neighbor images to each contrastive batch, reporting improved sensitivity to word order and binding. TripletCLIP uses instead generative AI models, generating a fluent hard negative caption with a large language model and a matching image with a text-to-image diffusion model, and training with a triplet contrastive objective.
• Reweighting in-batch negatives. DiHT use a hard-negative-weighted contrastive loss for image-text pretraining that up-weights each in-batch, uniformly-sampled negative in proportion to its similarity to the anchor, extending the importance-weighting approach to cross-modal alignment.
• Pair-level selection. HELIP proposes an algorithm to mine image-text hard negative pairs and adds a margin loss function that keeps mined hard negatives closer to the positive than ordinary negatives.

Embedding collapse from over-hard negatives

Several authors report that mining only the single hardest negative destabilizes embedding training. FaceNet found that always selecting the hardest negatives could drive the embedding to a collapsed solution, and adopted semi-hard mining instead. Lifted structured embedding similarly observed that selecting only the hardest negative converges to a poor local optimum, motivating a smoothed objective over all in-margin negatives. Distance-weighted sampling diagnoses the mechanism as noise-dominated, high-variance gradients, and an empirical comparison in hierarchical-triplet-loss work found naive hardest-negative mining giving little improvement over random sampling, with a semi-hard variant performing better, partly because mining within a mini-batch cannot capture the global data distribution.

False negatives

When labels are unavailable or incomplete, the hardest negatives are also the most likely to be false negatives: examples that are mined as negatives but actually share the anchor's class or relevance. The problem has been documented in unsupervised contrastive learning, where it is called sampling bias (not to be confused with sampling bias in statistics), and in dense retrieval, where one study estimated that a large fraction of top-retrieved but unlabelled passages were in fact relevant, so that mining them as negatives without denoising degraded retrieval quality. Proposed remedies include debiasing corrections, cross-encoder denoising, uncertainty- and diversity-aware selection, and, in multi-modal settings, using a third modality to filter likely false negatives before reweighting.