Activation Functions

Formula

Graph

Sigmoid

$g\left(x\right)=\frac{1}{1+{\text{e}}^{-x}}$

Tangent Sigmoid

$g\left(x\right)=\tanh \left(x\right)$

ReLU (rectified

linear unit)

$g\left(x\right)=\max \left(0,x\right)$