Tokenization
In plain terms
Breaking text into smaller units a model can process.
Going deeper
Tokenization splits input into tokens (words, subwords, or characters) that the model maps to numbers. How text is tokenized affects model behavior, costs, and context limits.
Under the hood
Subword schemes (BPE, WordPiece, SentencePiece) balance vocabulary size and handling of rare words; token count drives context limits and API cost.
Examples
GPT's BPE tokenizer, splitting 'unhappiness' into sub-tokens, ~4 chars per token rule of thumb.
Think of it like...
Like cutting a sentence into puzzle pieces small enough for the machine to handle one at a time.