OpenAI tokenizer playground

Paste text (or ids), then press Count.

• Counts are the BPE output of the installed gpt-tokenizer encoding — a port of OpenAI's tiktoken. Exact for these encodings: what the matching OpenAI models charge for is what is counted.
• Claude, Gemini, and other vendors use different tokenizers. Counts here are NOT their counts — the context-fit tool labels those as estimates instead of pretending.
• Special tokens pasted into the text (like <|endoftext|>) count as their single special id — that is what a model actually receives.
• A token can hold partial UTF-8 (an emoji often splits); its slice may show the replacement character. That is the real boundary, not a bug.

• Plain sentence

  Ordinary English — watch spaces lead most tokens.

• Emoji + accents

  Astral emoji and CJK carry multi-byte tokens; one grapheme is not one token.

• Code snippet

  Code tokens split on punctuation runs — why code costs more tokens than prose.

• Special token

  A pasted special token counts as its single special id — what the model actually sees.