kl_divergence#
- class braintools.metric.kl_divergence(log_predictions, targets)#
Compute the Kullback-Leibler divergence (relative entropy) loss.
KL divergence [1] measures the information lost when approximating the target distribution with the predicted distribution. It quantifies how much one probability distribution differs from another.
The KL divergence is defined as:
\[D_{KL}(P||Q) = \sum_i P(i) \log\frac{P(i)}{Q(i)}\]where P is the target distribution and Q is the predicted distribution.
- Parameters:
log_predictions (
Array|ndarray|bool|number|bool|int|float|complex|Quantity) – Log probabilities of the predicted distribution with shape[..., num_classes]. Must be in log-space to avoid numerical underflow issues.targets (
Array|ndarray|bool|number|bool|int|float|complex|Quantity) – Probabilities of the target distribution with shape[..., num_classes]. Values should be non-negative and sum to 1 along the last axis. Must be strictly positive where non-zero.
- Returns:
KL divergence values with shape
[...], measuring the divergence between target and predicted distributions.- Return type:
Array|ndarray|bool|number|bool|int|float|complex|Quantity
Examples
>>> import jax.numpy as jnp >>> import braintools >>> # Target and predicted distributions >>> targets = jnp.array([[0.7, 0.2, 0.1]]) >>> log_preds = jnp.log(jnp.array([[0.6, 0.3, 0.1]])) >>> kl_div = braintools.metric.kl_divergence(log_preds, targets) >>> print(f"KL divergence: {kl_div[0]:.4f}") KL divergence: 0.0268
Notes
KL divergence is not symmetric: KL(P||Q) ≠ KL(Q||P). It measures the information lost when using Q to approximate P. The function handles zero probabilities by setting the corresponding terms to zero.
References