Cross-Entropy Loss Surface

Cross-Entropy Loss Surface: experiment guide

This direct mathematical tool evaluates one labeled example from a three-class classifier. True class selects the target label, with exactly one of classes 0, 1, or 2 considered correct. P(class 0) and P(class 1) are editable probabilities between zero and one. The third probability is the remainder, 1 − P(class 0) − P(class 1). The fields are not independent unconstrained scores: their sum cannot exceed one. An impossible distribution produces an explicit error and no numerical loss, rather than silently clipping or renormalizing your inputs. Reset restores probabilities (0.5, 0.3, 0.2) and target class 0.

Worked details 1

The heatmap is a top-down view of the valid probability triangle, sampled on a fixed grid. Horizontal position denotes P(class 0), vertical position denotes P(class 1), and color encodes negative natural logarithm of the chosen target probability. Darker cells have larger loss; the visual color range saturates at 8 nats, while the textual loss remains uncapped. The uncolored region lies outside the simplex and is not a valid distribution. A cross marks your input position. Color is only a secondary cue: the table lists each probability and its target indicator, and the result states the exact formula, target probability, and loss. Changing True class recomputes the entire surface.

Worked details 2

With the reset distribution and class 0 selected, the expected loss is −ln(0.5), approximately 0.693147 nats. Select class 1 without changing the probabilities: the expected value is −ln(0.3), approximately 1.203973. Select class 2 to obtain −ln(0.2), approximately 1.609438. For the confident-wrong experiment, select class 0, set P(class 0) to 0.01 and P(class 1) to 0.3. Loss rises to approximately 4.605170. Set P(class 0) to 0.95 and P(class 1) to 0.03 to obtain approximately 0.051293. These observations follow from logarithms and are not empirical training measurements.

Worked details 3

The table also shows p minus the one-hot target, the gradient with respect to logits when these probabilities arise from softmax. It is not the derivative with respect to the probability fields themselves; that derivative is −1 divided by the target probability in the target coordinate. Logit-gradient entries sum to zero. At an exact zero probability, the displayed p minus target entries are limiting values, because a finite softmax logit cannot produce an exact zero. For a boundary test, set target class 2 and probabilities 0.5 and 0.5: loss is positive infinity. At a target probability of one it is zero. This lesson omits label smoothing, class weights, batch averaging, and numerical softmax implementations. It measures the mathematical surface directly, so adding animation would not make the calculation more faithful.