What is Token?
A token is the unit of text a language model reads and generates — roughly a word or word-fragment. Models price usage, measure context limits and bill throughput per token, so token count directly drives AI cost and latency.
Text is split into tokens before a model processes it (about 0.75 words per token in English). Understanding tokenisation matters in production because it determines API cost, how much fits in the context window, and where long inputs must be trimmed or retrieved.