Listwise Ranking Loss#

ranking_softmax_loss(logits, labels, *, where=None, weights=None, reduction=None, reduce_fn=jnp.mean) trains a model to score the items of a list so that relevant items come first. For one list with scores \(s\), relevance labels \(l \ge 0\) and optional weights \(w\) it computes

\[L = -\sum_i w_i\, l_i \log \operatorname{softmax}(s)_i ,\]

the cross-entropy between the labels and the softmax over the list (the ListNet / softmax loss). It matches optax.losses.ranking_softmax_loss. Two consequences of the formula:

  • labels are not normalized: with graded labels the loss scales with their sum, so the relative size of labels within a list sets how much each item matters;

  • weights multiply the labels, so an item with label 0 is unaffected by its weight.

The loss works over the last axis; leading axes are a batch of lists.

1. Setup#

import jax
import jax.numpy as jnp
import numpy as np
import optax

import braintools.metric as bm

2. One List#

scores = jnp.array([2.0, 1.0, 0.5, 0.2])
labels = jnp.array([1.0, 0.0, 0.0, 0.0])  # item 0 is relevant

loss = bm.ranking_softmax_loss(scores, labels)
print(f"loss {float(loss):.4f} = -log softmax(s)[0] = {float(-jax.nn.log_softmax(scores)[0]):.4f}")

graded = jnp.array([3.0, 1.0, 0.0, 0.0])
print(f"graded labels: {float(bm.ranking_softmax_loss(scores, graded)):.4f} "
      f"(by hand {float(-(graded * jax.nn.log_softmax(scores)).sum()):.4f}; "
      f"optax {float(optax.losses.ranking_softmax_loss(scores, graded)):.4f})")
loss 0.5632 = -log softmax(s)[0] = 0.5632
graded labels: 3.2529 (by hand 3.2529; optax 3.2529)

3. Batches of Variable-Length Lists#

Pad lists to a common length and pass a boolean mask where (same shape as the logits). Masked items get a logit of \(-\infty\) before the softmax, so they neither receive probability mass nor contribute to the loss.

The reduction over lists is chosen with reduction='mean' | 'sum' | 'none' (or the older reduce_fn). A list whose items are all masked contributes 0 and is left out of the mean.

logits = jnp.array([[2.0, 1.0, 0.5, -1.0, 0.0],
                    [0.8, 0.3, 1.2, -2.0, -1.0],
                    [0.1, 0.2, 0.3, 0.4, 0.5]])
labels = jnp.array([[1.0, 0.0, 0.0, 0.0, 0.0],
                    [0.0, 0.0, 1.0, 0.0, 0.0],
                    [1.0, 0.0, 0.0, 0.0, 0.0]])
where = jnp.array([[True, True, True, True, False],
                   [True, True, True, False, False],
                   [False, False, False, False, False]])  # an empty list

per_list = bm.ranking_softmax_loss(logits, labels, where=where, reduction="none")
print("per list:", per_list)
print("mean over non-empty lists:", bm.ranking_softmax_loss(logits, labels, where=where, reduction="mean"),
      "=", float(per_list[:2].mean()))
print("sum:", bm.ranking_softmax_loss(logits, labels, where=where, reduction="sum"))

# The padded item cannot change the loss:
changed = logits.at[0, 4].set(100.0)
print("padded logit 0 -> 100 changes list 0 by",
      float(bm.ranking_softmax_loss(changed, labels, where=where, reduction="none")[0] - per_list[0]))
per list: 
[0.49518192 0.73087144 0.        ]
mean over non-empty lists: 0.6130267 = 0.6130266785621643
sum: 1.2260534
padded logit 0 -> 100 changes list 0 by 0.0

4. Weights#

Weights scale each item’s label, so they matter only for items with a positive label. Doubling the weight of a list’s only relevant item doubles that list’s loss; weighting irrelevant items does nothing.

w_relevant = jnp.where(labels > 0, 2.0, 1.0)
w_irrelevant = jnp.where(labels > 0, 1.0, 5.0)
print("no weights:            ", bm.ranking_softmax_loss(logits, labels, where=where, reduction="none"))
print("relevant items x2:     ", bm.ranking_softmax_loss(logits, labels, where=where, weights=w_relevant, reduction="none"))
print("irrelevant items x5:   ", bm.ranking_softmax_loss(logits, labels, where=where, weights=w_irrelevant, reduction="none"))
no weights:             [0.49518192 0.73087144 0.        ]
relevant items x2:      [0.99036384 1.4617429  0.        ]
irrelevant items x5:    [0.49518192 0.73087144 0.        ]

5. Learning to Rank#

A linear scoring model trained with this loss learns to put the relevant items on top. Each query has 10 candidate items with 5 features; relevance is graded 0, 1, 2 from a hidden linear score. Quality is measured with the mean reciprocal rank (MRR) of the most relevant item.

rng = np.random.default_rng(0)
n_queries, n_items, n_feat = 200, 10, 5
features = jnp.asarray(rng.normal(size=(n_queries, n_items, n_feat)))
true_w = jnp.asarray(rng.normal(size=n_feat))
latent = features @ true_w
relevance = jnp.digitize(latent, jnp.quantile(latent, jnp.array([0.6, 0.9]))).astype(jnp.float32)  # 0, 1, 2


def loss_fn(w):
    return bm.ranking_softmax_loss(features @ w, relevance, reduction="mean")


def mrr(w):
    order = jnp.argsort(-(features @ w), axis=-1)
    best = jnp.argmax(latent, axis=-1)
    rank = jnp.argmax(order == best[:, None], axis=-1) + 1
    return float(jnp.mean(1.0 / rank))


w = jnp.zeros(n_feat)
grad = jax.jit(jax.grad(loss_fn))
print(f"before training: loss {float(loss_fn(w)):.3f}, MRR {mrr(w):.3f}")
for _ in range(300):
    w = w - 0.5 * grad(w)
cos = float(w @ true_w / jnp.linalg.norm(w) / jnp.linalg.norm(true_w))
print(f"after training:  loss {float(loss_fn(w)):.3f}, MRR {mrr(w):.3f}, cosine(w, true_w) {cos:.3f}")
before training: loss 11.513, MRR 0.289
after training:  loss 8.741, MRR 0.900, cosine(w, true_w) 0.982

Summary#

  • The loss is the cross-entropy \(-\sum_i w_i l_i \log\operatorname{softmax}(s)_i\) over the last axis, identical to optax’s.

  • Labels are not normalized; weights multiply labels.

  • where masks padding (masked items get \(-\infty\) logits); empty lists contribute 0 and are skipped by 'mean'.

  • Prefer reduction='none' | 'mean' | 'sum'; reduce_fn is kept for backward compatibility.