gelu

Contents

gelu#

class brainunit.math.gelu(x, approximate=True, unit_to_scale=None)#

Gaussian error linear unit activation function.

If approximate=False, computes the element-wise function:

\[\mathrm{gelu}(x) = \frac{x}{2} \left(\mathrm{erfc} \left( \frac{-x}{\sqrt{2}} \right) \right)\]

If approximate=True, uses the approximate formulation of GELU:

\[\mathrm{gelu}(x) = \frac{x}{2} \left(1 + \mathrm{tanh} \left( \sqrt{\frac{2}{\pi}} \left(x + 0.044715 x^3 \right) \right) \right)\]

For more information, see Gaussian Error Linear Units (GELUs), section 2.

Parameters:

  • x (Quantity | Array | ndarray | bool | number | bool | int | float | complex) – Input array. Must be unitless if a Quantity.

  • approximate (bool) – Whether to use the approximate or exact formulation. Default is True.

  • unit_to_scale (Unit | None) – Unit used to convert x to a dimensionless number before applying the activation.

Returns:

out – An array with the same shape as x.

Return type:

Array

Examples

>>> import jax.numpy as jnp
>>> import saiunit.math as sumath
>>> sumath.gelu(jnp.array([-2., 0., 2.]))
Array([-0.04540231,  0.        ,  1.9545977 ], dtype=float32)
Contents