PReLU

Contents

PReLU#

class brainstate.nn.PReLU(num_parameters=1, init=0.25, dtype=None)#

Applies the element-wise function.

\[\text{PReLU}(x) = \max(0,x) + a * \min(0,x)\]

or

\[\begin{split}\text{PReLU}(x) = \begin{cases} x, & \text{ if } x \geq 0 \\ ax, & \text{ otherwise } \end{cases}\end{split}\]

Here \(a\) is a learnable parameter. When called without arguments, nn.PReLU() uses a single parameter \(a\) across all input channels. If called with nn.PReLU(nChannels), a separate \(a\) is used for each input channel.

Parameters:

  • num_parameters (int) – Number of \(a\) to learn. Although it takes an int as input, there is only two values are legitimate: 1, or the number of channels at input. Default: 1

  • init (float) – The initial value of \(a\). Default: 0.25

  • dtype (optional) – The data type for the weight parameter.

  • Shape

  • -----

  • Input (-)

  • Output (-)

weight#

The learnable weights of shape (num_parameters).

Type:

Tensor

Notes

  • Weight decay should not be used when learning \(a\) for good performance.

  • Channel dim is the 2nd dim of input. When input has dims < 2, then there is no channel dim and the number of channels = 1.

Examples

>>> import brainstate
>>> m = brainstate.nn.PReLU()
>>> x = brainstate.random.randn(2)
>>> output = m(x)
Contents