15.1 Transformers

Transformers are the neural network architecture that powers nearly every modern large language model. Unlike older recurrent networks (RNNs) that read text one word at a time, a Transformer looks at an entire sequence at once and uses a mechanism called self-attention to decide which words matter most to each other.

Imagine translating an Amharic news headline into English. A Transformer can simultaneously weigh the subject, the verb and the object across the whole sentence, rather than forgetting the beginning by the time it reaches the end. This parallel processing is also why Transformers train so fast on modern GPUs.