Listwise Ranking Loss#
ranking_softmax_loss(logits, labels, *, where=None, weights=None, reduction=None, reduce_fn=jnp.mean) trains a model to score the items of a list so that relevant items come first. For one list with scores \(s\), relevance labels \(l \ge 0\) and optional weights \(w\) it computes
the cross-entropy between the labels and the softmax over the list (the ListNet / softmax loss). It matches optax.losses.ranking_softmax_loss. Two consequences of the formula:
labels are not normalized: with graded labels the loss scales with their sum, so the relative size of labels within a list sets how much each item matters;
weightsmultiply the labels, so an item with label 0 is unaffected by its weight.
The loss works over the last axis; leading axes are a batch of lists.
1. Setup#
import jax
import jax.numpy as jnp
import numpy as np
import optax
import braintools.metric as bm
2. One List#
scores = jnp.array([2.0, 1.0, 0.5, 0.2])
labels = jnp.array([1.0, 0.0, 0.0, 0.0]) # item 0 is relevant
loss = bm.ranking_softmax_loss(scores, labels)
print(f"loss {float(loss):.4f} = -log softmax(s)[0] = {float(-jax.nn.log_softmax(scores)[0]):.4f}")
graded = jnp.array([3.0, 1.0, 0.0, 0.0])
print(f"graded labels: {float(bm.ranking_softmax_loss(scores, graded)):.4f} "
f"(by hand {float(-(graded * jax.nn.log_softmax(scores)).sum()):.4f}; "
f"optax {float(optax.losses.ranking_softmax_loss(scores, graded)):.4f})")
loss 0.5632 = -log softmax(s)[0] = 0.5632
graded labels: 3.2529 (by hand 3.2529; optax 3.2529)
3. Batches of Variable-Length Lists#
Pad lists to a common length and pass a boolean mask where (same shape as the logits). Masked items get a logit of \(-\infty\) before the softmax, so they neither receive probability mass nor contribute to the loss.
The reduction over lists is chosen with reduction='mean' | 'sum' | 'none' (or the older reduce_fn). A list whose items are all masked contributes 0 and is left out of the mean.
logits = jnp.array([[2.0, 1.0, 0.5, -1.0, 0.0],
[0.8, 0.3, 1.2, -2.0, -1.0],
[0.1, 0.2, 0.3, 0.4, 0.5]])
labels = jnp.array([[1.0, 0.0, 0.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0, 0.0],
[1.0, 0.0, 0.0, 0.0, 0.0]])
where = jnp.array([[True, True, True, True, False],
[True, True, True, False, False],
[False, False, False, False, False]]) # an empty list
per_list = bm.ranking_softmax_loss(logits, labels, where=where, reduction="none")
print("per list:", per_list)
print("mean over non-empty lists:", bm.ranking_softmax_loss(logits, labels, where=where, reduction="mean"),
"=", float(per_list[:2].mean()))
print("sum:", bm.ranking_softmax_loss(logits, labels, where=where, reduction="sum"))
# The padded item cannot change the loss:
changed = logits.at[0, 4].set(100.0)
print("padded logit 0 -> 100 changes list 0 by",
float(bm.ranking_softmax_loss(changed, labels, where=where, reduction="none")[0] - per_list[0]))
per list:
[0.49518192 0.73087144 0. ]
mean over non-empty lists: 0.6130267 = 0.6130266785621643
sum: 1.2260534
padded logit 0 -> 100 changes list 0 by 0.0
4. Weights#
Weights scale each item’s label, so they matter only for items with a positive label. Doubling the weight of a list’s only relevant item doubles that list’s loss; weighting irrelevant items does nothing.
w_relevant = jnp.where(labels > 0, 2.0, 1.0)
w_irrelevant = jnp.where(labels > 0, 1.0, 5.0)
print("no weights: ", bm.ranking_softmax_loss(logits, labels, where=where, reduction="none"))
print("relevant items x2: ", bm.ranking_softmax_loss(logits, labels, where=where, weights=w_relevant, reduction="none"))
print("irrelevant items x5: ", bm.ranking_softmax_loss(logits, labels, where=where, weights=w_irrelevant, reduction="none"))
no weights: [0.49518192 0.73087144 0. ]
relevant items x2: [0.99036384 1.4617429 0. ]
irrelevant items x5: [0.49518192 0.73087144 0. ]
5. Learning to Rank#
A linear scoring model trained with this loss learns to put the relevant items on top. Each query has 10 candidate items with 5 features; relevance is graded 0, 1, 2 from a hidden linear score. Quality is measured with the mean reciprocal rank (MRR) of the most relevant item.
rng = np.random.default_rng(0)
n_queries, n_items, n_feat = 200, 10, 5
features = jnp.asarray(rng.normal(size=(n_queries, n_items, n_feat)))
true_w = jnp.asarray(rng.normal(size=n_feat))
latent = features @ true_w
relevance = jnp.digitize(latent, jnp.quantile(latent, jnp.array([0.6, 0.9]))).astype(jnp.float32) # 0, 1, 2
def loss_fn(w):
return bm.ranking_softmax_loss(features @ w, relevance, reduction="mean")
def mrr(w):
order = jnp.argsort(-(features @ w), axis=-1)
best = jnp.argmax(latent, axis=-1)
rank = jnp.argmax(order == best[:, None], axis=-1) + 1
return float(jnp.mean(1.0 / rank))
w = jnp.zeros(n_feat)
grad = jax.jit(jax.grad(loss_fn))
print(f"before training: loss {float(loss_fn(w)):.3f}, MRR {mrr(w):.3f}")
for _ in range(300):
w = w - 0.5 * grad(w)
cos = float(w @ true_w / jnp.linalg.norm(w) / jnp.linalg.norm(true_w))
print(f"after training: loss {float(loss_fn(w)):.3f}, MRR {mrr(w):.3f}, cosine(w, true_w) {cos:.3f}")
before training: loss 11.513, MRR 0.289
after training: loss 8.741, MRR 0.900, cosine(w, true_w) 0.982
Summary#
The loss is the cross-entropy \(-\sum_i w_i l_i \log\operatorname{softmax}(s)_i\) over the last axis, identical to optax’s.
Labels are not normalized; weights multiply labels.
wheremasks padding (masked items get \(-\infty\) logits); empty lists contribute 0 and are skipped by'mean'.Prefer
reduction='none' | 'mean' | 'sum';reduce_fnis kept for backward compatibility.