Token
Fundamentals Simplified version availableDefinition
The basic unit of text that AI language models process. A token is roughly equivalent to a word or part of a word. Models have a "context window" that limits how many tokens they can process at once.
Example in context
The sentence "AI is fascinating" contains four tokens — though exact tokenisation varies by model.
Related terms
Also explore
Financial DictionaryMissing a term?
Help us build the best AI dictionary for everyone.