How to handle rate limits¶
When you call the OpenAI API repeatedly, you may encounter error messages that say 429: 'Too Many Requests' or RateLimitError. These error messages come from exceeding the API's rate limits.
This guide shares tips for avoiding and handling rate limit errors.
To see an example script for throttling parallel requests to avoid rate limit errors, see api_request_parallel_processor.py.
Why rate limits exist¶
Rate limits are a common practice for APIs, and they're put in place for a few different reasons.
- First, they help protect against abuse or misuse of the API. For example, a malicious actor could flood the API with requests in an attempt to overload it or cause disruptions in service. By setting rate limits, OpenAI can prevent this kind of activity.
- Second, rate limits help ensure that everyone has fair access to the API. If one person or organization makes an excessive number of requests, it could bog down the API for everyone else. By throttling the number of requests that a single user can make, OpenAI ensures that everyone has an opportunity to use the API without experiencing slowdowns.
- Lastly, rate limits can help OpenAI manage the aggregate load on its infrastructure. If requests to the API increase dramatically, it could tax the servers and cause performance issues. By setting rate limits, OpenAI can help maintain a smooth and consistent experience for all users.
Although hitting rate limits can be frustrating, rate limits exist to protect the reliable operation of the API for its users.
Default rate limits¶
As of Jan 2023, the default rate limits are:
| Text Completion & Embedding endpoints | Code & Edit endpoints | |
|---|---|---|
| Free trial users |
|
|
| Pay-as-you-go users (in your first 48 hours) |
|
|
| Pay-as-you-go users (after your first 48 hours) |
|
|
For reference, 1,000 tokens is roughly a page of text.
Other rate limit resources¶
Read more about OpenAI's rate limits in these other resources:
- Guide: Rate limits
- Help Center: Is API usage subject to any rate limits?
- Help Center: How can I solve 429: 'Too Many Requests' errors?
Requesting a rate limit increase¶
If you'd like your organization's rate limit increased, please fill out the following form:
Example rate limit error¶
A rate limit error will occur when API requests are sent too quickly. If using the OpenAI Python library, they will look something like:
RateLimitError: Rate limit reached for default-codex in organization org-{id} on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues or if you’d like to request an increase.
Below is example code for triggering a rate limit error.
import openai # for making OpenAI API requests
# request a bunch of completions in a loop
for _ in range(100):
openai.Completion.create(
model="code-cushman-001",
prompt="def magic_function():\n\t",
max_tokens=10,
)
How to avoid rate limit errors¶
Retrying with exponential backoff¶
One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached.
This approach has many benefits:
- Automatic retries means you can recover from rate limit errors without crashes or missing data
- Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail
- Adding random jitter to the delay helps retries from all hitting at the same time
Note that unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work.
Below are a few example solutions.
Example #1: Using the Tenacity library¶
Tenacity is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.
To add exponential backoff to your requests, you can use the tenacity.retry decorator. The following example uses the tenacity.wait_random_exponential function to add random exponential backoff to a request.
Note that the Tenacity library is a third-party tool, and OpenAI makes no guarantees about its reliability or security.
import openai # for OpenAI API calls
from tenacity import (
retry,
stop_after_attempt,
wait_random_exponential,
) # for exponential backoff
@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
def completion_with_backoff(**kwargs):
return openai.Completion.create(**kwargs)
completion_with_backoff(model="text-davinci-002", prompt="Once upon a time,")
<OpenAIObject text_completion id=cmpl-5oowO391reUW8RGVfFyzBM1uBs4A5 at 0x10d8cae00> JSON: {
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"text": " a little girl dreamed of becoming a model.\n\nNowadays, that dream"
}
],
"created": 1662793900,
"id": "cmpl-5oowO391reUW8RGVfFyzBM1uBs4A5",
"model": "text-davinci-002",
"object": "text_completion",
"usage": {
"completion_tokens": 16,
"prompt_tokens": 5,
"total_tokens": 21
}
}
import backoff # for exponential backoff
import openai # for OpenAI API calls
@backoff.on_exception(backoff.expo, openai.error.RateLimitError)
def completions_with_backoff(**kwargs):
return openai.Completion.create(**kwargs)
completions_with_backoff(model="text-davinci-002", prompt="Once upon a time,")
<OpenAIObject text_completion id=cmpl-5oowPhIdUvshEsF1rBhhwE9KFfI3M at 0x111043680> JSON: {
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"text": " two children lived in a poor country village. In the winter, the temperature would"
}
],
"created": 1662793901,
"id": "cmpl-5oowPhIdUvshEsF1rBhhwE9KFfI3M",
"model": "text-davinci-002",
"object": "text_completion",
"usage": {
"completion_tokens": 16,
"prompt_tokens": 5,
"total_tokens": 21
}
}
Example 3: Manual backoff implementation¶
If you don't want to use third-party libraries, you can implement your own backoff logic.
# imports
import random
import time
import openai
# define a retry decorator
def retry_with_exponential_backoff(
func,
initial_delay: float = 1,
exponential_base: float = 2,
jitter: bool = True,
max_retries: int = 10,
errors: tuple = (openai.error.RateLimitError,),
):
"""Retry a function with exponential backoff."""
def wrapper(*args, **kwargs):
# Initialize variables
num_retries = 0
delay = initial_delay
# Loop until a successful response or max_retries is hit or an exception is raised
while True:
try:
return func(*args, **kwargs)
# Retry on specified errors
except errors as e:
# Increment retries
num_retries += 1
# Check if max retries has been reached
if num_retries > max_retries:
raise Exception(
f"Maximum number of retries ({max_retries}) exceeded."
)
# Increment the delay
delay *= exponential_base * (1 + jitter * random.random())
# Sleep for the delay
time.sleep(delay)
# Raise exceptions for any errors not specified
except Exception as e:
raise e
return wrapper
@retry_with_exponential_backoff
def completions_with_backoff(**kwargs):
return openai.Completion.create(**kwargs)
completions_with_backoff(model="text-davinci-002", prompt="Once upon a time,")
<OpenAIObject text_completion id=cmpl-5oowRsCXv3AkUgVJyyo3TQrVq7hIT at 0x111024220> JSON: {
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"text": " a man decided to greatly improve his karma by turning his life around.\n\n"
}
],
"created": 1662793903,
"id": "cmpl-5oowRsCXv3AkUgVJyyo3TQrVq7hIT",
"model": "text-davinci-002",
"object": "text_completion",
"usage": {
"completion_tokens": 16,
"prompt_tokens": 5,
"total_tokens": 21
}
}
How to maximize throughput of batch processing given rate limits¶
If you're processing real-time requests from users, backoff and retry is a great strategy to minimize latency while avoiding rate limit errors.
However, if you're processing large volumes of batch data, where throughput matters more than latency, there are a few other things you can do in addition to backoff and retry.
Proactively adding delay between requests¶
If you are constantly hitting the rate limit, then backing off, then hitting the rate limit again, then backing off again, it's possible that a good fraction of your request budget will be 'wasted' on requests that need to be retried. This limits your processing throughput, given a fixed rate limit.
Here, one potential solution is to calculate your rate limit and add a delay equal to its reciprocal (e.g., if your rate limit 20 requests per minute, add a delay of 3–6 seconds to each request). This can help you operate near the rate limit ceiling without hitting it and incurring wasted requests.
Example of adding delay to a request¶
# imports
import time
import openai
# Define a function that adds a delay to a Completion API call
def delayed_completion(delay_in_seconds: float = 1, **kwargs):
"""Delay a completion by a specified amount of time."""
# Sleep for the delay
time.sleep(delay_in_seconds)
# Call the Completion API and return the result
return openai.Completion.create(**kwargs)
# Calculate the delay based on your rate limit
rate_limit_per_minute = 20
delay = 60.0 / rate_limit_per_minute
delayed_completion(
delay_in_seconds=delay,
model="text-davinci-002",
prompt="Once upon a time,"
)
<OpenAIObject text_completion id=cmpl-5oowVVZnAzdCPtUJ0rifeamtLcZRp at 0x11b2c7680> JSON: {
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"text": " there was an idyllic little farm that sat by a babbling brook"
}
],
"created": 1662793907,
"id": "cmpl-5oowVVZnAzdCPtUJ0rifeamtLcZRp",
"model": "text-davinci-002",
"object": "text_completion",
"usage": {
"completion_tokens": 16,
"prompt_tokens": 5,
"total_tokens": 21
}
}
Batching requests¶
The OpenAI API has separate limits for requests per minute and tokens per minute.
If you're hitting the limit on requests per minute, but have headroom on tokens per minute, you can increase your throughput by batching multiple tasks into each request. This will allow you to process more tokens per minute, especially with the smaller models.
Sending in a batch of prompts works exactly the same as a normal API call, except that pass in a list of strings to prompt parameter instead of a single string.
Warning: the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the index field.
Example without batching¶
import openai # for making OpenAI API requests
num_stories = 10
prompt = "Once upon a time,"
# serial example, with one story completion per request
for _ in range(num_stories):
response = openai.Completion.create(
model="curie",
prompt=prompt,
max_tokens=20,
)
# print story
print(prompt + response.choices[0].text)
Once upon a time, before there were grandiloquent tales of the massacre at Fort Mims, there were stories of Once upon a time, a full-sized search and rescue was created. However, CIDIs are the addition of requiring Once upon a time, Schubert was hot with the films. “Schubert sings of honey, flowers, Once upon a time, you could watch these films on your VCR, sometimes years after their initial theatrical release, and there Once upon a time, there was a forest. In that forest, the forest animals ruled. The forest animals had their homes Once upon a time, there were two programs that complained about false positive scans. Peacock and Midnight Manager alike, only Once upon a time, a long, long time ago, tragedy struck. it was the darkest of nights, and there was Once upon a time, when Adam was a perfect little gentleman, he was presented at Court as a guarantee of good character. Once upon a time, Adam and Eve made a mistake. They ate the fruit from the tree of immortality and split the consequences Once upon a time, there was a set of programming fundamental principles known as the "X model." This is a set of
Example with batching¶
import openai # for making OpenAI API requests
num_stories = 10
prompts = ["Once upon a time,"] * num_stories
# batched example, with 10 stories completions per request
response = openai.Completion.create(
model="curie",
prompt=prompts,
max_tokens=20,
)
# match completions to prompts by index
stories = [""] * len(prompts)
for choice in response.choices:
stories[choice.index] = prompts[choice.index] + choice.text
# print stories
for story in stories:
print(story)
Once upon a time, there were two sisters, Eliza Pickering and Ariana 'Ari' Lucas. When these lovely Once upon a time, Keene was stung by a worm — actually, probably a python — snaking through his leg Once upon a time, there was a professor of physics during the depression. It was difficult, during this time, to get Once upon a time, before you got sick, you told stories to all and sundry, and your listeners believed in you Once upon a time, there was one very old nice donkey. He was incredibly smart, in a very old, kind of Once upon a time, the property of a common lodging house was a common cup for all the inhabitants. Betimes a constant Once upon a time, in an unspecified country, there was a witch who had an illegal product. It was highly effective, Once upon a time, a long time ago, I turned 13, my beautiful dog Duncan swept me up into his jaws like Once upon a time, as a thoroughly reformed creature from an army of Nazis, he took On Judgement Day myself and his Once upon a time, Capcom made a game for the Atari VCS called Missile Command. While it was innovative at the time
Example parallel processing script¶
We've written an example script for parallel processing large quantities of API requests: api_request_parallel_processor.py.
The script combines some handy features:
- Streams requests from file, to avoid running out of memory for giant jobs
- Makes requests concurrently, to maximize throughput
- Throttles both request and token usage, to stay under rate limits
- Retries failed requests, to avoid missing data
- Logs errors, to diagnose problems with requests
Feel free to use it as is or modify it to suit your needs.