GELU

Contents

GELU#

class brainstate.nn.GELU(approximate=False)#

Applies the Gaussian Error Linear Units function.

\[\text{GELU}(x) = x * \Phi(x)\]

where \(\Phi(x)\) is the Cumulative Distribution Function for Gaussian Distribution.

When the approximate argument is True, Gelu is estimated with:

\[\text{GELU}(x) = 0.5 * x * (1 + \text{Tanh}(\sqrt(2 / \pi) * (x + 0.044715 * x^3)))\]
Parameters:

  • approximate (bool) – Whether to use the tanh approximation algorithm. Default: False

  • Shape

  • -----

  • Input (-)

  • Output (-)

Examples

>>> import brainstate.nn as nn
>>> import brainstate
>>> m = nn.GELU()
>>> x = brainstate.random.randn(2)
>>> output = m(x)
Contents