What is Tokenisation?
Verified 5 August 2026
Straightforward for space-delimited languages and genuinely hard otherwise. Chinese, Japanese and Thai write without spaces, so a tokeniser must either segment statistically or fall back to character n-grams.
Character bigrams are robust: they need no language-specific model, they degrade gracefully on mixed-script text, and the same rule applies at index and query time so the two always agree.
Common questions
What is Tokenisation?
Tokenisation splits text into indexable units — usually words, but character n-grams for scripts without word delimiters.
How does Tokenisation work in practice?
Character bigrams are robust: they need no language-specific model, they degrade gracefully on mixed-script text, and the same rule applies at index and query time so the two always agree.
See it working
The free tier is 10,000 requests a month with no card — enough to test any of this against your own queries.