Skip to main content
Dev Proxy

Simulate throttling of the OpenAI API

Last modified on February 5, 2025

This sample contains a preset and mocks to simulate the throttling of the OpenAI API. As more and more applications use OpenAI API to use AI, it's important that developers can verify that their apps can properly handle cases when the API is throttled. Especially when the app is using lower-tier API plans, throttling is more likely to happen so validating the app's behavior is crucial to ensure great user experience. Using this preset you can simulate throttling of the OpenAI API and see how your app will handle it. The preset includes 3 different throttling scenarios: exceeded tokens per minute, exceeded number of requests per minute and service overload, which includes dynamic simulation of handling retry-after. Proxy will simulate throttling the API using one of these modes at random.

View on GitHub Download from GitHub

References

Get started with the Dev Proxy

The tutorial will introduce you to the Dev Proxy and show you how to use its features.

Change mocks file

Instructions on how to change the mocks file used by the Dev Proxy.

Use preset configurations

Instructions on how to configure the Dev Proxy to use a different configuration file.

OpenAI API reference

The OpenAI API reference documentation.