How to count tokens with tiktoken¶

tiktoken is a fast open-source tokenizer by OpenAI.

Given a text string (e.g., "tiktoken is great!") and an encoding (e.g., "cl100k_base"), a tokenizer can split the text string into a list of tokens (e.g., ["t", "ik", "token", " is", " great", "!"]).

Splitting text strings into tokens is useful because GPT models see text in the form of tokens. Knowing how many tokens are in a text string can tell you (a) whether the string is too long for a text model to process and (b) how much an OpenAI API call costs (as usage is priced by token). Different models use different encodings.

Encodings¶

Encodings specify how text is converted into tokens. Different models use different encodings.

tiktoken supports three encodings used by OpenAI models:

Encoding name	OpenAI models
cl100k_base	ChatGPT models, text-embedding-ada-002
p50k_base	Code models, text-davinci-002, text-davinci-003
r50k_base (or gpt2)	GPT-3 models like davinci

You can retrieve the encoding for a model using tiktoken.encoding_for_model() as follows:

encoding = tiktoken.encoding_for_model('gpt-3.5-turbo')

p50k_base overlaps substantially with r50k_base, and for non-code applications, they will usually give the same tokens.

Tokenizer libraries by language¶

For cl100k_base and p50k_base encodings, tiktoken is the only tokenizer available as of March 2023.

• Python: tiktoken

For r50k_base (gpt2) encodings, tokenizers are available in many languages.

• Python: tiktoken (or alternatively GPT2TokenizerFast)
• JavaScript: gpt-3-encoder
• .NET / C#: GPT Tokenizer
• Java: gpt2-tokenizer-java
• PHP: GPT-3-Encoder-PHP

(OpenAI makes no endorsements or guarantees of third-party libraries.)

How strings are typically tokenized¶

In English, tokens commonly range in length from one character to one word (e.g., "t" or " great"), though in some languages tokens can be shorter than one character or longer than one word. Spaces are usually grouped with the starts of words (e.g., " is" instead of "is " or " "+"is"). You can quickly check how a string is tokenized at the OpenAI Tokenizer.

0. Install tiktoken¶

Install tiktoken with pip:

In [1]:

%pip install --upgrade tiktoken

%pip install --upgrade tiktoken

Requirement already satisfied: tiktoken in /Users/ted/.virtualenvs/openai/lib/python3.9/site-packages (0.3.2)
Requirement already satisfied: regex>=2022.1.18 in /Users/ted/.virtualenvs/openai/lib/python3.9/site-packages (from tiktoken) (2022.10.31)
Requirement already satisfied: requests>=2.26.0 in /Users/ted/.virtualenvs/openai/lib/python3.9/site-packages (from tiktoken) (2.28.2)
Requirement already satisfied: idna<4,>=2.5 in /Users/ted/.virtualenvs/openai/lib/python3.9/site-packages (from requests>=2.26.0->tiktoken) (3.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/ted/.virtualenvs/openai/lib/python3.9/site-packages (from requests>=2.26.0->tiktoken) (2.0.9)
Requirement already satisfied: certifi>=2017.4.17 in /Users/ted/.virtualenvs/openai/lib/python3.9/site-packages (from requests>=2.26.0->tiktoken) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/ted/.virtualenvs/openai/lib/python3.9/site-packages (from requests>=2.26.0->tiktoken) (1.26.7)
Note: you may need to restart the kernel to use updated packages.

1. Import tiktoken¶

In [2]:

import tiktoken

import tiktoken

2. Load an encoding¶

Use tiktoken.get_encoding() to load an encoding by name.

The first time this runs, it will require an internet connection to download. Later runs won't need an internet connection.

In [3]:

encoding = tiktoken.get_encoding("cl100k_base")

encoding = tiktoken.get_encoding("cl100k_base")

Use tiktoken.encoding_for_model() to automatically load the correct encoding for a given model name.

In [4]:

encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")

encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")

3. Turn text into tokens with encoding.encode()¶

The .encode() method converts a text string into a list of token integers.

In [5]:

encoding.encode("tiktoken is great!")

encoding.encode("tiktoken is great!")

Out[5]:

[83, 1609, 5963, 374, 2294, 0]

Count tokens by counting the length of the list returned by .encode().

In [6]:

def num_tokens_from_string(string: str, encoding_name: str) -> int:
    """Returns the number of tokens in a text string."""
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

def num_tokens_from_string(string: str, encoding_name: str) -> int: """Returns the number of tokens in a text string.""" encoding = tiktoken.get_encoding(encoding_name) num_tokens = len(encoding.encode(string)) return num_tokens

In [7]:

num_tokens_from_string("tiktoken is great!", "cl100k_base")

num_tokens_from_string("tiktoken is great!", "cl100k_base")

Out[7]:

6

4. Turn tokens into text with encoding.decode()¶

.decode() converts a list of token integers to a string.

In [8]:

encoding.decode([83, 1609, 5963, 374, 2294, 0])

encoding.decode([83, 1609, 5963, 374, 2294, 0])

Out[8]:

'tiktoken is great!'

Warning: although .decode() can be applied to single tokens, beware that it can be lossy for tokens that aren't on utf-8 boundaries.

For single tokens, .decode_single_token_bytes() safely converts a single integer token to the bytes it represents.

In [9]:

[encoding.decode_single_token_bytes(token) for token in [83, 1609, 5963, 374, 2294, 0]]

[encoding.decode_single_token_bytes(token) for token in [83, 1609, 5963, 374, 2294, 0]]

Out[9]:

[b't', b'ik', b'token', b' is', b' great', b'!']

(The b in front of the strings indicates that the strings are byte strings.)

5. Comparing encodings¶

Different encodings can vary in how they split words, group spaces, and handle non-English characters. Using the methods above, we can compare different encodings on a few example strings.

In [10]:

def compare_encodings(example_string: str) -> None:
    """Prints a comparison of three string encodings."""
    # print the example string
    print(f'\nExample string: "{example_string}"')
    # for each encoding, print the # of tokens, the token integers, and the token bytes
    for encoding_name in ["gpt2", "p50k_base", "cl100k_base"]:
        encoding = tiktoken.get_encoding(encoding_name)
        token_integers = encoding.encode(example_string)
        num_tokens = len(token_integers)
        token_bytes = [encoding.decode_single_token_bytes(token) for token in token_integers]
        print()
        print(f"{encoding_name}: {num_tokens} tokens")
        print(f"token integers: {token_integers}")
        print(f"token bytes: {token_bytes}")

def compare_encodings(example_string: str) -> None: """Prints a comparison of three string encodings.""" # print the example string print(f'\nExample string: "{example_string}"') # for each encoding, print the # of tokens, the token integers, and the token bytes for encoding_name in ["gpt2", "p50k_base", "cl100k_base"]: encoding = tiktoken.get_encoding(encoding_name) token_integers = encoding.encode(example_string) num_tokens = len(token_integers) token_bytes = [encoding.decode_single_token_bytes(token) for token in token_integers] print() print(f"{encoding_name}: {num_tokens} tokens") print(f"token integers: {token_integers}") print(f"token bytes: {token_bytes}")

In [11]:

compare_encodings("antidisestablishmentarianism")

compare_encodings("antidisestablishmentarianism")

Example string: "antidisestablishmentarianism"

gpt2: 5 tokens
token integers: [415, 29207, 44390, 3699, 1042]
token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism']

p50k_base: 5 tokens
token integers: [415, 29207, 44390, 3699, 1042]
token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism']

cl100k_base: 6 tokens
token integers: [519, 85342, 34500, 479, 8997, 2191]
token bytes: [b'ant', b'idis', b'establish', b'ment', b'arian', b'ism']

In [12]:

compare_encodings("2 + 2 = 4")

compare_encodings("2 + 2 = 4")

Example string: "2 + 2 = 4"

gpt2: 5 tokens
token integers: [17, 1343, 362, 796, 604]
token bytes: [b'2', b' +', b' 2', b' =', b' 4']

p50k_base: 5 tokens
token integers: [17, 1343, 362, 796, 604]
token bytes: [b'2', b' +', b' 2', b' =', b' 4']

cl100k_base: 7 tokens
token integers: [17, 489, 220, 17, 284, 220, 19]
token bytes: [b'2', b' +', b' ', b'2', b' =', b' ', b'4']

In [13]:

compare_encodings("お誕生日おめでとう")

compare_encodings("お誕生日おめでとう")

Example string: "お誕生日おめでとう"

gpt2: 14 tokens
token integers: [2515, 232, 45739, 243, 37955, 33768, 98, 2515, 232, 1792, 223, 30640, 30201, 29557]
token bytes: [b'\xe3\x81', b'\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f', b'\xe6\x97', b'\xa5', b'\xe3\x81', b'\x8a', b'\xe3\x82', b'\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8', b'\xe3\x81\x86']

p50k_base: 14 tokens
token integers: [2515, 232, 45739, 243, 37955, 33768, 98, 2515, 232, 1792, 223, 30640, 30201, 29557]
token bytes: [b'\xe3\x81', b'\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f', b'\xe6\x97', b'\xa5', b'\xe3\x81', b'\x8a', b'\xe3\x82', b'\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8', b'\xe3\x81\x86']

cl100k_base: 9 tokens
token integers: [33334, 45918, 243, 21990, 9080, 33334, 62004, 16556, 78699]
token bytes: [b'\xe3\x81\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f', b'\xe6\x97\xa5', b'\xe3\x81\x8a', b'\xe3\x82\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8\xe3\x81\x86']

6. Counting tokens for chat API calls¶

ChatGPT models like gpt-3.5-turbo use tokens in the same way as past completions models, but because of their message-based formatting, it's more difficult to count how many tokens will be used by a conversation.

Below is an example function for counting tokens for messages passed to gpt-3.5-turbo-0301 or gpt-4-0314.

Note that the exact way that messages are converted into tokens may change from model to model. So when future model versions are released, the answers returned by this function may be only approximate. The ChatML documentation explains in more detail how the OpenAI API converts messages into tokens.

In [14]:

def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0301"):
    """Returns the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        print("Warning: model not found. Using cl100k_base encoding.")
        encoding = tiktoken.get_encoding("cl100k_base")
    if model == "gpt-3.5-turbo":
        print("Warning: gpt-3.5-turbo may change over time. Returning num tokens assuming gpt-3.5-turbo-0301.")
        return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0301")
    elif model == "gpt-4":
        print("Warning: gpt-4 may change over time. Returning num tokens assuming gpt-4-0314.")
        return num_tokens_from_messages(messages, model="gpt-4-0314")
    elif model == "gpt-3.5-turbo-0301":
        tokens_per_message = 4  # every message follows <im_start>{role/name}\n{content}<im_end>\n
        tokens_per_name = -1  # if there's a name, the role is omitted
    elif model == "gpt-4-0314":
        tokens_per_message = 3
        tokens_per_name = 1
    else:
        raise NotImplementedError(f"""num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens.""")
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 2  # every reply is primed with <im_start>assistant
    return num_tokens

def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0301"): """Returns the number of tokens used by a list of messages.""" try: encoding = tiktoken.encoding_for_model(model) except KeyError: print("Warning: model not found. Using cl100k_base encoding.") encoding = tiktoken.get_encoding("cl100k_base") if model == "gpt-3.5-turbo": print("Warning: gpt-3.5-turbo may change over time. Returning num tokens assuming gpt-3.5-turbo-0301.") return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0301") elif model == "gpt-4": print("Warning: gpt-4 may change over time. Returning num tokens assuming gpt-4-0314.") return num_tokens_from_messages(messages, model="gpt-4-0314") elif model == "gpt-3.5-turbo-0301": tokens_per_message = 4 # every message follows {role/name}\n{content}\n tokens_per_name = -1 # if there's a name, the role is omitted elif model == "gpt-4-0314": tokens_per_message = 3 tokens_per_name = 1 else: raise NotImplementedError(f"""num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens.""") num_tokens = 0 for message in messages: num_tokens += tokens_per_message for key, value in message.items(): num_tokens += len(encoding.encode(value)) if key == "name": num_tokens += tokens_per_name num_tokens += 2 # every reply is primed with assistant return num_tokens

In [15]:

# let's verify the function above matches the OpenAI API response

import openai

example_messages = [
    {
        "role": "system",
        "content": "You are a helpful, pattern-following assistant that translates corporate jargon into plain English.",
    },
    {
        "role": "system",
        "name": "example_user",
        "content": "New synergies will help drive top-line growth.",
    },
    {
        "role": "system",
        "name": "example_assistant",
        "content": "Things working well together will increase revenue.",
    },
    {
        "role": "system",
        "name": "example_user",
        "content": "Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.",
    },
    {
        "role": "system",
        "name": "example_assistant",
        "content": "Let's talk later when we're less busy about how to do better.",
    },
    {
        "role": "user",
        "content": "This late pivot means we don't have time to boil the ocean for the client deliverable.",
    },
]

for model in ["gpt-3.5-turbo-0301", "gpt-4-0314"]:
    print(model)
    # example token count from the function defined above
    print(f"{num_tokens_from_messages(example_messages, model)} prompt tokens counted by num_tokens_from_messages().")
    # example token count from the OpenAI API
    response = openai.ChatCompletion.create(
        model=model,
        messages=example_messages,
        temperature=0,
        max_tokens=1  # we're only counting input tokens here, so let's not waste tokens on the output
    )
    print(f'{response["usage"]["prompt_tokens"]} prompt tokens counted by the OpenAI API.')
    print()

# let's verify the function above matches the OpenAI API response import openai example_messages = [ { "role": "system", "content": "You are a helpful, pattern-following assistant that translates corporate jargon into plain English.", }, { "role": "system", "name": "example_user", "content": "New synergies will help drive top-line growth.", }, { "role": "system", "name": "example_assistant", "content": "Things working well together will increase revenue.", }, { "role": "system", "name": "example_user", "content": "Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.", }, { "role": "system", "name": "example_assistant", "content": "Let's talk later when we're less busy about how to do better.", }, { "role": "user", "content": "This late pivot means we don't have time to boil the ocean for the client deliverable.", }, ] for model in ["gpt-3.5-turbo-0301", "gpt-4-0314"]: print(model) # example token count from the function defined above print(f"{num_tokens_from_messages(example_messages, model)} prompt tokens counted by num_tokens_from_messages().") # example token count from the OpenAI API response = openai.ChatCompletion.create( model=model, messages=example_messages, temperature=0, max_tokens=1 # we're only counting input tokens here, so let's not waste tokens on the output ) print(f'{response["usage"]["prompt_tokens"]} prompt tokens counted by the OpenAI API.') print()

gpt-3.5-turbo-0301
126 prompt tokens counted by num_tokens_from_messages().
126 prompt tokens counted by the OpenAI API.

gpt-4-0314
128 prompt tokens counted by num_tokens_from_messages().
128 prompt tokens counted by the OpenAI API.