12.6 Activation Functions

An activation function decides the output of a neuron after the weighted sum and bias. Crucially, it introduces nonlinearity, which is what lets a stack of layers model complex, curved patterns instead of just straight lines. Common choices include ReLU, sigmoid, tanh, and softmax.

Think of a water tap with a valve: below a certain pressure nothing flows, then flow increases. ReLU behaves similarly — it outputs zero for negative inputs and passes positive inputs straight through, which is simple and fast to compute.