14.2 Tokenization

Tokenization is the process of splitting text into smaller pieces called tokens. A token can be a word, a sub-word, or even a single character. Models cannot read raw sentences; they need text broken into consistent units that can each be mapped to a number.

Consider the Amharic sentence 'ኢትዮጵያ ቆንጆ ናት'. A simple word tokenizer splits it into ['ኢትዮጵያ', 'ቆንጆ', 'ናት']. Modern systems often use sub-word tokenization, which can split a rare word into known pieces so the model still understands part of it.