15.2 Attention Mechanism

The attention mechanism is the heart of a Transformer. For every token, the model creates three vectors: a Query (what am I looking for?), a Key (what do I offer?) and a Value (what information do I carry?). It compares each Query against all Keys to decide how much attention to pay to each token, then blends the Values accordingly.

Consider the sentence 'The cup fell because it was full.' To understand what 'it' refers to, attention lets the model link 'it' strongly to 'cup' rather than 'because'. In multi-head attention, several such comparisons run in parallel, each capturing a different kind of relationship.