Why architecture matters here
The reason to care is that raw cross-entropy optimizes for a target the model can never actually reach, and chasing the unreachable does damage. A softmax output equals 1.0 only when its logit is infinitely larger than the rest, so a loss that wants the true class at 1.0 keeps pushing the winning logit up and the losing logits down without bound. Past the point of correct classification, this extra pushing does not improve accuracy — the argmax is already right — it just inflates confidence. The model ends up producing probabilities that are systematically too extreme: it says 99.9% when it should say 90%, and when it is wrong it is wrong with high confidence. For any system that consumes the probabilities — thresholding, ranking, abstaining, calibrated ensembling, risk-weighted decisions — that miscalibration is a direct correctness problem, not a cosmetic one.
Label smoothing addresses this at the source: it changes the target so the optimal solution is a finite, confident-but-not-infinite logit configuration. Because the true class's target is 1-ε rather than 1.0, and each other class has a small positive target ε/(K-1), the gradient no longer pushes the winning logit toward infinity — it pushes toward a specific finite gap where the predicted distribution matches the soft target. The model still learns to be confident (with a small ε it is still assigning the vast majority of mass to the correct class) but the confidence is capped, which is precisely what fixes calibration. The regularization is not an external penalty added to the loss; it is baked into the target the loss compares against.
The second reason to care is representational. When you require the true class logit to sit a bounded distance above the others rather than infinitely above, and you require the non-true logits to be roughly equal (since they all share the same small target), you encourage the penultimate-layer features to form tight clusters that are equidistant from the wrong-class directions. Empirically this shows up as cleaner, more separated class representations, which is part of why label smoothing often improves accuracy and not merely calibration — it is a genuine regularizer that constrains the geometry of the learned features, similar in spirit to weight decay or dropout but acting through the target rather than the weights or activations.
The trade-off is that the softened targets deliberately discard some information, and in one important setting that information matters. Knowledge distillation trains a student on the teacher's full soft output — the relative probabilities the teacher assigns to wrong classes carry the 'dark knowledge' about class similarity that makes distillation work. Label smoothing on the teacher erases exactly those relative differences (it forces the wrong classes toward a uniform small value), so a teacher trained with label smoothing is a worse teacher even if it is a better-calibrated classifier. Recognizing when the suppressed information is valuable versus harmful is the architectural judgment the technique demands.
The architecture: every piece explained
Top row: the target transformation. The one-hot target places all mass on the true class. The smoothing parameter ε (a small value like 0.1) is the fraction of mass moved off the true class and spread over the rest. The soft target is the result: the true class gets 1-ε, and each of the other K-1 classes gets ε/(K-1). This is a mixture of the original one-hot distribution and a uniform distribution, weighted (1-ε) and ε. Cross-entropy is then computed between the model's predicted distribution and this soft target instead of the hard one — the only change to the training objective, but a structural one.
Middle row: how it flows through the model's output. The model produces logits; softmax turns them into a predicted distribution; cross-entropy against the soft target produces a gradient that pulls the prediction toward 1-ε on the true class and toward ε/(K-1) on the others. The critical consequence is the bounded logit gap: because the target for the true class is strictly less than 1, the gradient reaches zero at a finite logit separation rather than pushing forever. There is a stable equilibrium — the model is done learning when its predicted distribution matches the soft target — instead of an unreachable target it chases into overconfidence.
Bottom rows: the downstream effects and the ops surface. Calibration improves because the capped confidence means predicted probabilities track empirical accuracy more closely — a model that says 90% is right about 90% of the time, which is the definition of a well-calibrated classifier. The representation effect is the tighter, more equidistant class clustering in the feature space that the soft target induces. The ops strip names the practical decisions: choosing ε (too small does nothing, too large over-smooths and hurts accuracy; 0.1 is a common default), watching calibration with reliability diagrams and expected calibration error rather than accuracy alone, disabling it for distillation where soft logits are the point, and evaluating both metrics because label smoothing trades a little of one for a lot of the other and you must measure which.
One precise point about the loss form: with smoothing, cross-entropy against the soft target can be decomposed into the original cross-entropy against the true class plus a term that is the KL divergence (up to a constant) between the uniform distribution and the model's prediction. That second term is the regularizer — it penalizes the model for deviating from uniform on the non-true classes, which is what caps confidence. Seeing the loss this way makes clear that label smoothing is cross-entropy plus a confidence penalty, not a mysterious new objective, and it explains why the effect scales with ε: ε is literally the weight on the uniform-matching regularization term.
End-to-end flow
Trace one training step for a classifier with K classes and ε = 0.1. A sample arrives whose true class is index 2. Without smoothing the target would be [0, 0, 1, 0, ...]. With smoothing, the target becomes 1-ε = 0.9 at index 2 and ε/(K-1) at every other index — a soft distribution that still overwhelmingly favors class 2 but leaves a little mass everywhere else.
The model produces logits, softmax turns them into a predicted distribution, and cross-entropy is computed against the soft target. Early in training the model's prediction is far from the target, so the gradient is large and pushes the class-2 logit up and the others down — the same direction as ordinary training. The difference emerges as the model gets confident. With a hard target, when the model already predicts, say, 0.99 for class 2, the loss still wants 1.0, so the gradient keeps pushing the logit higher. With the soft target, once the model predicts 0.9 for class 2 and the small remainder for the rest, the prediction matches the target and the gradient goes to zero. The logit gap stops growing at a finite value. The model has learned to be confident to exactly the degree the soft target specifies and no more.
Now see the two downstream effects concretely. On calibration: because the logit gap is bounded, the model's peak probabilities are capped near 1-ε rather than at 0.9999, so when you plot predicted confidence against actual accuracy (a reliability diagram), the curve sits much closer to the diagonal — the model's stated confidence is trustworthy. On representation: because every wrong class shares the same small target, the network is discouraged from making any one wrong class's logit special, which pushes the feature representations toward tight clusters equidistant from the wrong-class boundaries. Both effects come from the same single change to the target.
The performance character is a trade you can measure. Label smoothing usually leaves top-1 accuracy the same or slightly better while substantially improving calibration — a very favorable exchange for any system that uses the probabilities downstream. But the improvement is not free of tension: because the model is trained never to be more than 1-ε confident, its raw output probabilities are systematically 'less sharp,' which is fine for decisions but is exactly wrong if something downstream depends on the fine structure of the full distribution. The clearest case is temperature-scaled ensembling or distillation, where the shape of the tail matters.
Consider that stress case directly: you train a large teacher with label smoothing because it gives the teacher great calibration, then distill it into a small student. Distillation works by having the student match the teacher's soft probabilities over all classes — the teacher's assignment of, say, 3% to 'truck' when the answer is 'car' tells the student that cars and trucks are similar, and that relational 'dark knowledge' is what makes the student learn efficiently. But label smoothing deliberately flattened those wrong-class probabilities toward a uniform ε/(K-1), erasing the very structure distillation depends on. The student learns worse from the smoothed teacher than from an un-smoothed one, even though the smoothed teacher is the better standalone classifier. The lesson is that label smoothing improves the model's decisions at the cost of the information content of its full output distribution, and whether that trade is right depends entirely on whether anything downstream consumes that full distribution — which is a design decision, not a default.