15.3 GPT Architecture

GPT (Generative Pre-trained Transformer) is a 'decoder-only' Transformer. It is trained on a simple but powerful objective: given a stretch of text, predict the next token. By repeating this prediction over and over, GPT generates fluent text one token at a time, feeding each new token back in as context.

Think of how your phone keyboard suggests the next word when you type 'I am going to...'. GPT does the same thing but at a vastly larger scale, having read enormous amounts of text. That is why it can draft an email in Amharic or explain a recipe — it is predicting plausible continuations.