GLU

Contents

GLU#

class brainstate.nn.GLU(dim=-1)#

Applies the gated linear unit function.

\[{GLU}(a, b)= a \otimes \sigma(b)\]

where \(a\) is the first half of the input matrices and \(b\) is the second half.

Parameters:

  • dim (int) – The dimension on which to split the input. Default: -1

  • Shape

  • -----

  • Input (-) – dimensions

  • Output (-)

Examples

>>> import brainstate.nn as nn
>>> import brainstate
>>> m = nn.GLU()
>>> x = brainstate.random.randn(4, 2)
>>> output = m(x)
Contents