GPT-4 token encoder and decoder

This example shows how to use a language model tokenizer to encode and decode and how to search for a token

Note

Install pandas and tiktoken with pip install pandas tiktoken

This is an educational tool for understanding how tokenization works.

Enter text to tokenize it:

 13617Example

 1495 text

 374 is

 1618 here

Or convert space separated tokens to text:

Search for a token:

50,257 results