gelu

Contents

gelu#

class brainstate.nn.gelu(x, approximate=True)[source]#

Gaussian Error Linear Unit activation function.

If approximate=False, computes the element-wise function:

\[\mathrm{gelu}(x) = \frac{x}{2} \left(1 + \mathrm{erf} \left( \frac{x}{\sqrt{2}} \right) \right)\]

If approximate=True, uses the approximate formulation of GELU:

\[\mathrm{gelu}(x) = \frac{x}{2} \left(1 + \mathrm{tanh} \left( \sqrt{\frac{2}{\pi}} \left(x + 0.044715 x^3 \right) \right) \right)\]
Parameters:

  • x (Array | ndarray | bool | number | bool | int | float | complex | Quantity) – Input array.

  • approximate (bool) – Whether to use the approximate (True) or exact (False) formulation. Default is True.

Returns:

An array with the same shape as the input.

Return type:

Array | Quantity

References

Contents