Running on your device · 0 bytes uploaded

OpenAI tokenizer playground

Count tokens exactly with the encoding a model actually uses — every token id and slice listed, both directions, computed in a worker.
no Web Workers in this environment — exact counts need them; the page stays honest insteadCountClearText → tokensToken ids → textEncoding

Paste text (or ids), then press Count.

- Counts are the BPE output of the installed gpt-tokenizer encoding — a port of OpenAI's tiktoken. Exact for these encodings: what the matching OpenAI models charge for is what is counted.
- Claude, Gemini, and other vendors use different tokenizers. Counts here are NOT their counts — the context-fit tool labels those as estimates instead of pretending.
- Special tokens pasted into the text (like <|endoftext|>) count as their single special id — that is what a model actually receives.
- A token can hold partial UTF-8 (an emoji often splits); its slice may show the replacement character. That is the real boundary, not a bug.

- Plain sentence

Ordinary English — watch spaces lead most tokens.

- Emoji + accents

Astral emoji and CJK carry multi-byte tokens; one grapheme is not one token.

- Code snippet

Code tokens split on punctuation runs — why code costs more tokens than prose.

- Special token

A pasted special token counts as its single special id — what the model actually sees.

Nothing in yet

Paste, drop, or type to begin. Everything stays on this device.

nothing counts until you press Count

Next door in AI

- Context fit (/ai/context-fit)
- API cost estimator (/ai/cost-estimator)
All AI tools (/ai)

Canonical HTML: https://nutter.tools/ai/tokenizer
Markdown version: https://nutter.tools/ai/tokenizer/index.md
Plain-text version: https://nutter.tools/ai/tokenizer/index.txt
Agent index: https://nutter.tools/llms.txt