Cauchy prior regularization.
Implements regularization based on the negative log-likelihood of a
Cauchy distribution (Student’s t with df=1), which has very heavy tails:
\[L = \lambda \left[ \sum_i \log\left(1 + (x_i / s)^2\right) + N \log s \right]\]
where \(N\) is the number of elements and \(N \log s\) is the scale
log-normalizer that keeps the loss bounded below when scale is trained.
- Parameters:
weight (float) – Regularization weight (lambda). Default is 1.0.
scale (float) – Scale parameter. Default is 1.0.
fit_hyper (bool) – Whether to optimize hyperparameters. Default is False.
Examples
>>> import jax.numpy as jnp
>>> from brainstate.nn import CauchyReg
>>> reg = CauchyReg(weight=1.0, scale=1.0)
>>> value = jnp.array([0.5, 5.0, -1.0])
>>> loss = reg.loss(value)
Notes
Cauchy prior allows for very large parameter values, making it extremely
robust but also allowing outliers. It has no defined mean or variance.
-
loss(value)[source]
Calculate Cauchy regularization loss.
- Parameters:
value (Array | ndarray | bool | number | bool | int | float | complex | Quantity) – Parameter values.
- Returns:
Cauchy negative log-likelihood loss.
- Return type:
Array | ndarray | bool | number | bool | int | float | complex | Quantity
-
reset_value()[source]
Return zero (the mode of symmetric Cauchy).
- Returns:
Zero.
- Return type:
Array | ndarray | bool | number | bool | int | float | complex | Quantity
-
sample_init(shape)[source]
Sample from Cauchy distribution.
- Parameters:
shape (int | Sequence[int] | integer | Sequence[integer]) – Shape of the sample.
- Returns:
Sample from Cauchy distribution.
- Return type:
Array | ndarray | bool | number | bool | int | float | complex | Quantity
