Azure OpenAI Batch API for Global Deployments

The Azure OpenAI Batch API is a game-changer for global deployments, allowing you to scale your applications quickly and efficiently.

With the Azure OpenAI Batch API, you can deploy AI models to multiple regions simultaneously, reducing latency and improving user experience.

This API supports up to 100 concurrent requests, making it ideal for large-scale deployments.

To get started, you'll need to create an Azure OpenAI account and enable the Batch API, which can be done in just a few clicks.

To get started with the Azure OpenAI Batch API, you'll need to meet a few prerequisites.

First, you'll need an Azure subscription, which you can create for free if you don't already have one.

You'll also need an Azure OpenAI resource with a Global-Batch model deployed. You can refer to the resource creation and model deployment guide for help with this process.

To ensure you're working with the latest version, you might need to upgrade your OpenAI Python library installation.

Store your API key securely, such as in Azure Key Vault, and never include it directly in your code or post it publicly.

Here's a quick rundown of the prerequisites:

You'll also need Python 3.8 or later version installed on your machine.

The Azure OpenAI Batch API is a game-changer for handling large, asynchronous groups of requests. It allows you to save a lot on costs, with a 50% cost reduction compared to the standard global pricing.

By bundling requests into a JSON lines (JSONL) file, you can process very large workloads more efficiently. This is especially useful for tasks that require a high volume of requests, such as natural language processing or text analysis.

Batch requests have their own enqueued token quotas, which means they don't interfere with real-time workloads. This ensures that your production workloads remain unaffected by the batch processing.

To get started with the Azure OpenAI Batch API, you'll need to create an Azure account and get an API key. You'll also need to install the langchain-openai integration package and deploy an Azure OpenAI instance.

To configure the Azure OpenAI Batch API, you need to create a resource group and a workspace in the Azure portal.

The Azure OpenAI Batch API uses a managed service, which allows you to focus on developing your application without worrying about the underlying infrastructure.

To deploy the Azure OpenAI Batch API, you can use the Azure CLI or Azure PowerShell, both of which provide a simple and efficient way to create and manage resources in Azure.

Global deployment is a powerful feature that allows you to deploy your AI models globally. This means your model will be available for use worldwide, making it a great option for applications that require global reach.

To enable global deployment, you'll need to choose the "Global-Batch" deployment type in the AI Foundry portal. This will give you access to a global queue of requests, where your model will be processed and executed.

One important thing to keep in mind is that dynamic quota is highly recommended for global batch model deployments. This will help prevent job failures due to insufficient enqueued token quota. With dynamic quota, your deployment can take advantage of extra quota when available, ensuring that your model can process requests smoothly.

Content filtering is also fully supported with global batch deployment, just like with other deployment types. You can create content filters and associate them with your global batch deployment for added flexibility.

The number of requests you can queue using batch is determined by your enqueued token quota. This quota includes the maximum number of input tokens you can enqueue at one time. Once your batch request is completed, your batch rate limit is reset, and your input tokens are cleared.

Here's a summary of the batch quota limits for different models:

Note: B = billion, M = million, K = thousand.

To create your batch file, you'll want to use Azure's JSON lines (.jsonl) format. Each line represents an individual request.

The JSONL file is a basic text file with a .jsonl extension. It contains details such as the method, url, and body for each request.

Here's a basic example of a JSONL file:

{"custom_id": "123", "method": "POST", "url": "https://example.com/api/endpoint", "body": {"model": "example_model", "messages": [{"role": "user", "content": "Hello, world!"}]}}

{"custom_id": "456", "method": "POST", "url": "https://example.com/api/endpoint", "body": {"model": "example_model", "messages": [{"role": "system", "content": "Welcome!"}]}}

To ensure your JSONL file is formatted correctly, you can reference the following parameters:

To use the Azure OpenAI Batch API, you'll need to format your input correctly. The API accepts input in four different formats: standard input, Base64 encoded image, image URL, and structured outputs.

You'll need to include a custom ID in your input to match the response you receive. Responses won't be returned in the same order as they were in the input file.

The model attribute must match the name of the Global Batch deployment you're targeting for inference responses. This ensures that the correct model is used for processing.

If you want to target a different deployment, you'll need to create a separate batch file or job. This helps prevent confusion and ensures that the correct model is used for each deployment.

For optimal performance, it's recommended to submit large files for batch processing rather than multiple small files. This can help speed up processing times and reduce errors.

Monitoring your Azure OpenAI Batch API jobs is crucial for ensuring they run smoothly and efficiently. You can monitor the progress of your batch job in Azure AI Studio, where you'll find detailed timestamps and status messages to track each phase of the job.

Azure provides a clear picture of your job's status, including possible values such as validating, failed, in_progress, finalizing, completed, expired, cancelling, and cancelled.

If your job fails, you'll receive error messages that can guide you through troubleshooting. The number of requests processed, pending, and failures that occurred are also easily accessible.

It's recommended to wait at least 60 seconds for each status call when monitoring via code. This ensures you get an accurate picture of your job's status.

You can also cancel any Batch call by running a specific command.

A successful job in the Azure OpenAI Batch API is marked by a Completed status, but it still generates an error_file_id associated with an empty file of zero bytes.

If a job fails, you'll find details about the failure in the errors property.

To identify the issue, check the error code, which will indicate the type of error that occurred. For example, if the error code is invalid_json_line, it means a line (or multiple lines) in your input file couldn't be parsed as valid JSON.

Here's a list of common error codes and their definitions:

By understanding these common error codes, you can quickly identify and resolve issues in your Azure OpenAI Batch API jobs.

Troubleshooting is an essential part of working with Azure OpenAI. A job is considered successful when its status is Completed, but even then, it may generate an error_file_id associated with an empty file with zero bytes.

If a job fails, you'll find details about the failure in the errors property. This can help you identify the root cause of the issue and take corrective action.

One common error code is invalid_json_line, which occurs when a line in your input file can't be parsed as valid JSON. This might be due to typos, missing brackets, or quotes not being used correctly. Resubmitting the request with the correct JSON formatting usually resolves this issue.

Another potential issue is too_many_tasks, which happens when the number of requests in your input file exceeds the maximum allowed value of 100,000. To fix this, simply reduce the total number of requests and resubmit the job.

Here are some common error codes and their definitions:

If you're using Azure OpenAI, there's a known issue with resources deployed using the Azure CLI. This can cause problems with Azure OpenAI global batch.

Resources deployed with Azure CLI might not work out-of-box with Azure OpenAI global batch due to an issue with endpoint subdomains.

A workaround for this issue is to deploy a new Azure OpenAI resource using one of the other common deployment methods, which will properly handle the subdomain setup as part of the deployment process.