A Comprehensive Guide to Azure OpenAI Architecture and Implementation

Azure OpenAI architecture offers a scalable and secure framework for building AI-powered applications. This architecture is designed to integrate with various Azure services, such as Azure Kubernetes Service (AKS) and Azure Storage.

By leveraging Azure OpenAI, developers can create complex AI models and deploy them in a production-ready environment. The architecture also provides a range of tools and services for data ingestion, processing, and storage.

Azure OpenAI supports multiple programming languages, including Python, C#, and Java. This flexibility allows developers to choose the language that best suits their project's requirements.

The Azure OpenAI architecture is built with several key components that work together to provide a robust and secure solution. Azure OpenAI is a fully managed service that provides REST API access to Azure OpenAI's language models, including GPT-4 and GPT-3.5-Turbo.

Azure AI Studio is a platform used to build, test, and deploy AI solutions, including the prompt flow orchestration logic for the chat application. This platform provides a managed virtual network for network security.

The architecture also includes Application Gateway, a layer 7 (HTTP/S) load balancer and web traffic router that distributes incoming traffic across availability zones and offloads encryption to improve application performance. Web Application Firewall (WAF) provides visibility into traffic to and from the web application, enabling you to monitor and secure your application.

Here are some key components of the Azure OpenAI architecture:

The components of this architecture are built upon the basics of Azure OpenAI, which provides REST API access to language models like GPT-4 and GPT-3.5-Turbo.

Azure AI Studio is used to build, test, and deploy AI solutions, including the prompt flow orchestration logic for the chat application, and provides a managed virtual network for network security.

Application Gateway is a layer 7 (HTTP/S) load balancer and web traffic router, which uses URL path-based routing to distribute incoming traffic across availability zones and offloads encryption to improve application performance.

A Web Application Firewall (WAF) protects web apps from common exploits like SQL injection and cross-site scripting, providing visibility into traffic to and from the web application.

Azure Key Vault securely stores and manages secrets, encryption keys, and certificates, centralizing the management of sensitive information.

Azure virtual network enables the creation of isolated and secure private virtual networks in Azure, necessary for a web application on App Service to use private endpoints for network-secure communication between resources.

Private Link allows clients to access Azure PaaS services directly from private virtual networks without using public IP addressing.

Azure DNS is a hosting service for DNS domains that provides name resolution using Microsoft Azure infrastructure, with private DNS zones mapping a service's FQDN to a private endpoint's IP address.

Here are the key components of this architecture at a glance:

Built-in Prompt Runtimes are a key component of this architecture, simplifying compute management and delegating most of the prompt flow configuration to the application's requirements.txt file and flow.dag.yaml configuration.

The Automatic Runtime is a serverless compute option that minimizes operational burden, making it a low maintenance choice. This approach is ideal for applications with straightforward requirements.

Using the Automatic Runtime, you can delegate most of the prompt flow configuration to the running application, eliminating the need for manual compute management. This results in an ephemeral and application-driven architecture.

However, for workloads with more complex requirements, an externalized compute, such as Compute Instance Runtime, may be necessary. In these cases, a more workload team-managed lifecycle of the compute is required.

Azure OpenAI's architecture prioritizes reliability and security to ensure a robust and compliant AI ecosystem. Establishing benchmarks with established security frameworks and compliance standards at an early stage is crucial for a secure deployment.

To maintain ongoing vigilance, regular risk assessments are conducted, and potential threats are proactively mitigated. This is achieved through continuous monitoring and a governance group focused on security to oversee and guide AI initiatives.

By implementing automated controls, such as Infrastructure as Code (IaC) practices, security controls are streamlined and efficiency is enhanced. Additionally, Azure OpenAI employs FIPS 140-2 compliant 256-bit AES encryption as a default data protection measure, providing an extra layer of security.

Azure OpenAI integrates seamlessly with Microsoft's robust security services, adhering to industry best practices and global standards like ISO 27001, HIPAA, and FedRAMP. This integration offers a multitude of significant advantages, including security ecosystem integration, compliance standards, and role-based access control.

Here are some key security features of Azure OpenAI:

Reliability Challenges are a key concern when it comes to workload reliability. Without a gateway, all reliability concerns must be addressed exclusively by using client logic and Azure OpenAI Service features.

Client logic and Azure OpenAI Service features alone are not enough to ensure workload reliability. This is because workload reliability depends on several factors, including its capacity for self-preservation and self-recovery, which are often implemented through replication and failover mechanisms.

Redundancy is a major challenge, as failing over between multiple Azure OpenAI instances based on service availability is a client responsibility that needs to be controlled through configuration and custom logic. This can be a complex task, especially when updating multiple client configurations for new Azure OpenAI instances.

Scale out to handle spikes is another challenge, as failing over to Azure OpenAI instances with capacity when throttled is also a client responsibility. This requires careful configuration and custom logic to direct low priority requests to a queue during high demand periods.

Load balancing or throttling is also a challenge, as Azure OpenAI APIs throttle requests by returning an HTTP 429 error response code to requests that exceed the Token-Per-Minute (TPM) or Requests-Per-Minute (RPM) in the pay-as-you-go model. To handle this, client implementations need to implement appropriate back-off and retry logic.

To mitigate these challenges, it's essential to monitor the required throughput in terms of tokens per minute (TPM) and requests per minute (RPM) and ensure that sufficient TPM is assigned from your quota to meet the demand for your deployments.

Data encryption is a crucial aspect of ensuring the security of your data. Azure OpenAI employs FIPS 140-2 compliant 256-bit AES encryption as a default data protection measure.

This means that your data is automatically encrypted, eliminating the need for user intervention. The encryption process is seamless and secure.

Custom key management options are also available, allowing you to take control of your encryption keys. Integration with Azure Key Vault provides an additional layer of security.

With these robust encryption measures in place, you can have peace of mind knowing your data is protected.

Azure OpenAI has achieved a FedRAMP High P-ATO in US Commercial regions, meeting the specific security requirements of US federal agencies and the Department of Defense (DoD) for certain data classifications.

This approval is a significant milestone, allowing government clients to securely access Azure OpenAI's services.

Government clients can opt to create a secure link between Azure Government and Azure OpenAI's Commercial Service, as described in the reference Architecture.

This connection enables secure and compliant use of Azure OpenAI's services by government agencies.

Establishing a secure deployment is crucial for any Azure OpenAI solution. Adhering to established security frameworks and compliance standards from the early stages of your project is essential.

Start by aligning your project with security frameworks like ISO 27001, HIPAA, and FedRAMP. This will help you set a strong foundation for your AI security journey.

Classifying data is also vital. Implement a systematic approach to categorize data in accordance with governance protocols, ensuring proper handling and protection.

Develop comprehensive AI governance rules and policies that cover the entire lifecycle of your AI solutions. This will help you stay on top of security and compliance.

To establish a strong security foundation, make full use of Microsoft's security features, including Identity and Access Management (IDAM), encryption, and network security.

Here are some key security best practices to keep in mind:

To manage identities and access in Azure OpenAI architecture, create separate managed identities for Azure AI Studio and Machine Learning resources. This includes creating a separate managed identity for each project and online endpoint.

Implement identity-access controls for the chat UI by using Microsoft Entra ID. This will help isolate different prompt flows from each other from a permissions perspective.

Give prompt flow authors access to only the projects they require, and make sure to onboard users to Azure AI Studio projects with least privilege role assignments.

To manage Azure Key Vault access, consider deploying a Key Vault instance specifically for the Azure AI Studio hub, and other Azure Key Vault instances as appropriate for other parts of your workload. This will help limit the scope of the Azure AI Studio managed identity's access to secrets, keys, and certificates.

Here are some key role assignments to keep in mind:

Identity Management is crucial to ensure secure and controlled access to Azure AI Studio resources. It's essential to create separate managed identities for different Azure AI Studio and Machine Learning resources.

Create separate managed identities for each project and online endpoint to isolate prompt flows from others. This will help prevent unauthorized access to sensitive data.

Implement identity-access controls for the chat UI using Microsoft Entra ID. This will ensure that only authorized users can access the chat UI.

To onboard users to Azure AI Studio projects, make least privilege role assignments to the resources they require. This will prevent users from accessing more resources than necessary.

The Azure AI Studio managed identity requires both control plane and data plane role assignments. This means it has read and write access to all secrets, keys, and certificates stored in the Azure key vault.

Consider deploying a Key Vault instance specifically for the Azure AI Studio hub, and other Azure Key Vault instances as appropriate for other parts of your workload. This will help segregate access to sensitive data.

Here are some best practices for identity management in Azure AI Studio:

Using Azure API Management for your gateway implementation is generally the preferred approach to building and operating an Azure OpenAI gateway. It's a platform as a service (PaaS) offering with rich built-in capabilities, high availability, and networking options.

Azure API Management is a platform-managed service designed to offload cross-cutting concerns for HTTP-based APIs. It supports customization through its inbound and outbound request processing policy system.

Most of the gateway routing and request handling logic must be implemented in the policy system of API Management. You can combine built-in policies and your own custom policies to achieve the desired functionality.

The GenAI gateway toolkit GitHub repository contains a number of custom API Management policies along with a load-testing setup for testing the behavior of the policies.

Using Azure API Management for your gateway implementation is generally the preferred approach to building and operating an Azure OpenAI gateway. It's preferred because the service is a PaaS offering with rich built-in capabilities, high availability, and networking options.

Here are some key benefits of using Azure API Management:

If your workload exists as part of an application landing zone, review the guidance available in the Cloud Adoption Framework for Azure on implementing an Azure API Management landing zone.

Logging and Bring Your Own Data (BYOD) are two key features in Azure OpenAI that work together to provide a robust management and identity solution.

Logging helps track and monitor AI service usage, providing insights into user activity and security.

You can use logging and BYOD together to enable detailed logging for compliance and security, as well as the flexibility to use your own data for custom AI applications.

Logging offers audit trails, incident response, accountability, and real-time alerts for enhanced compliance and security.

BYOD provides data control, customization, privacy, encryption, custom security protocols, and data classification tailored to specific organizational needs.

These features complement each other, allowing you to have a secure and customizable AI solution that meets your organization's unique requirements.

The shared hosting, or consumption, model can be subject to noisy neighbor or other stressors on the platform, making it less predictable than provisioned throughput.

To ensure performance efficiency, it's essential to monitor provision-managed utilization for provisioned throughput. This will help you understand how your application is utilizing reserved processing capacity.

A gateway service can introduce a throughput bottleneck, so ensure it has adequate performance to handle full concurrent load and can easily scale in line with your growth expectations.

To evaluate your design, use the Design review checklist for Performance Efficiency, which considers factors such as throughput bottleneck, processing latency, and geographical proximity to users and Azure OpenAI instances.

Here are some key considerations for networking and performance in Azure OpenAI architecture:

Networking is crucial for any computer system. A network can be as simple as two computers connected with a cable or as complex as a global network of interconnected servers.

A network's performance can be affected by the type of network topology used. For example, a star topology, where all devices connect to a central hub, is more efficient than a bus topology, where devices connect to a single cable.

Network congestion can occur when too many devices are connected to a network, causing a decrease in performance. This is often seen in large networks with many users.

A good network infrastructure is essential for optimal performance. This includes using high-quality cables and routers, and ensuring that all devices are compatible with each other.

Network protocols, such as TCP/IP, help devices communicate with each other across a network. Understanding these protocols is essential for troubleshooting network issues.

Regular network maintenance, such as updating firmware and checking for software updates, can help prevent network downtime and ensure optimal performance.

Performance Efficiency is all about maintaining a smooth user experience even when there's an increase in load by managing capacity. This involves scaling resources, identifying and optimizing potential bottlenecks, and optimizing for peak performance.

To start your design strategy, use the design review checklist for Performance Efficiency to define a baseline based on key performance indicators for Azure OpenAI workloads.

Capacity is a crucial aspect of Performance Efficiency. To estimate consumers' elasticity demands, identify high-priority traffic that requires synchronous responses and low-priority traffic that can be asynchronous and batched.

Benchmark token consumption requirements based on estimated demands from consumers. Consider using the Azure OpenAI benchmarking tool to help you validate the throughput if you're using provisioned throughput unit (PTU) deployments.

Provisioned throughput offers dedicated memory and compute, reserved capacity, and consistent maximum latency for the specified model version. However, the pay-as-you-go offering can suffer from noisy neighbor problems like increased latency and throttling in regions under heavy use.

To ensure performance efficiency, add the appropriate gateways in front of your Azure OpenAI deployments. Ensure that the gateway can route to multiple instances in the same or different regions.

Here's a list of key considerations for capacity:

Cost optimization is crucial for any Azure OpenAI architecture, and it starts with identifying unnecessary expenses and improving operational efficiencies.

To reduce costs, you can use the Azure pricing calculator to customize a pricing example for your specific usage. The most expensive components in a typical scenario are DDoS Protection and the firewall deployed as part of the managed online endpoint.

You can optimize resources such as chat UI and prompt flow compute and AI Search to save the most cost.

Provisioned throughput is a billing model that ensures reserved processing capacity for your OpenAI model deployments, providing predictable performance and throughput for your models. This is unlike the shared hosting, or consumption, model, which is best-effort and might be subject to noisy neighbor or other stressors on the platform.

Monitor provision-managed utilization for provisioned throughput to ensure you're getting the most out of your resources.

To achieve performance efficiency, you can follow the guidance on autoscaling an online endpoint to remain closely aligned with demand without excessive overprovisioning, especially in low-usage periods.

Choosing the right virtual machine SKU for the online endpoint is also crucial to meet your performance targets. Test the performance of both lower instance count and bigger SKUs versus larger instance count and smaller SKUs to find an optimal configuration.

Here are some key considerations for performance efficiency:

When considering an API gateway, ensure it has adequate performance to handle full concurrent load and can easily scale in line with your growth expectations.

To deploy the Azure OpenAI architecture, you'll need to follow the steps in the OpenAI end-to-end baseline reference implementation.

You can't simply plug in a turn-key solution to proxy Azure OpenAI's HTTP API or other custom language model inferencing APIs, but a gateway in Azure is a viable option for your workload team to implement.

This means you'll need to get creative and find a solution that works for your specific needs.

Operational Excellence is about setting up procedures for development practices, observability, and release management to ensure your workload runs smoothly.

To start, you should ensure deployment of Azure OpenAI instances across various environments like development, test, and production. This allows for continuous learning and experimentation throughout the development cycle.

Monitoring and visualizing metrics is key to observability, and you can do this by using Azure Diagnostics. This tool collects and analyzes metrics and logs, helping you monitor availability, performance, and operation of Azure OpenAI.

If Azure OpenAI diagnostics aren't enough, consider using a gateway like Azure API Management to log incoming prompts and outgoing responses. This information can help you understand the effectiveness of the model for incoming prompts.

To deploy with confidence, use infrastructure as code (IaC) to deploy Azure OpenAI, model deployments, and other infrastructure required for fine-tuning models. This ensures consistency and reliability in your deployments.

Here's a checklist to help you achieve Operational Excellence with Azure OpenAI:

Remember to follow large language model operations (LLMOps) practices to operationalize the management of your Azure OpenAI LLMs, including deployment, fine-tuning, and prompt engineering.

To deploy and run this scenario, follow the steps in the OpenAI end-to-end baseline reference implementation. This is the recommended approach for deploying the reference implementation.

You can find the steps to deploy the reference implementation in the OpenAI documentation. It's a straightforward process that will get you up and running quickly.

If you're looking for alternative implementation options, Azure doesn't offer a turn-key solution designed specifically for this purpose. However, a gateway in Azure can be used as a viable alternative.

To get started, select the subscription where your new resources will be located. This is an important step as it will determine where your resources will be housed.

You'll then need to create or select a resource group. Don't worry too much about this step, as you can always create a new resource group later if needed.

Set the region for your resource carefully, as not every model type is available in every region. For access to gpt-4o models, you need to select East US 2 at the moment of writing this post.

After filling in these details, you can leave other values as default and use the Next button to proceed.

An API gateway can be a viable alternative, but it introduces new considerations into your architecture, such as evaluating the workload impact across these architectural considerations.

Carefully evaluate whether the added value or utility of the gateway justifies the added complexity, as every new component introduced into your architecture brings its own set of tradeoffs.

Azure services offer alternatives to hosting prompt flow, such as other compute services that may better align with your workload's functional and nonfunctional requirements.

You may want to consider hosting your workload on an alternative compute, especially if you're looking for better alignment with your functional and nonfunctional requirements.

An example of why this is a consideration is discussed later in the article, highlighting the potential benefits of exploring alternative options.

Azure services can offer more suitable alternatives to hosting prompt flow, allowing you to choose the best fit for your specific needs.

These alternatives can help you achieve a better balance between your workload's functional and nonfunctional requirements, potentially leading to improved performance and efficiency.

You can explore these alternatives to find the best solution for your workload, taking into account its unique needs and requirements.

Dealing with security challenges can be a real headache. One major issue is that all security concerns must be addressed in client logic and Azure OpenAI Service features, which can be limiting.

Identity management is a key area of concern. Authentication scope is a problem because Azure OpenAI instance-level authentication introduces complexity for providing least privileged access and identity segmentation.

Identity providers can also be an issue. Clients that can't use identities located in the Microsoft Entra tenant must share a single full-access API key, which is not ideal.

Network security is another challenge. Depending on client location relative to Azure OpenAI instances, public internet access to language models might be necessary.

Data sovereignty is also a concern. Your workload needs to ensure regional affinity to comply with data residency and sovereignty laws, which can involve multiple Azure OpenAI deployments.

Here are some specific security challenges to consider:

Introducing a new component into your architecture can have significant tradeoffs. This means you need to carefully evaluate the pros and cons of making changes.

A key consideration is the workload impact of introducing a new component. This is especially true when injecting an API gateway between clients and the Azure OpenAI data plane.

The added value or utility of the gateway should be weighed against the workload impact. This is a crucial step in determining whether the gateway is a good solution for your architecture.

In some cases, the workload impact may not be justified by the benefits of the gateway. This means you need to consider alternative solutions that meet your needs without introducing unnecessary complexity.

Ultimately, the decision to introduce a new component depends on a careful evaluation of the tradeoffs involved.

Azure OpenAI sets itself apart from other OpenAI services by seamlessly integrating with Microsoft's robust security services and adhering to industry best practices.

One of the key advantages of this integration is the Security Ecosystem, which combines OpenAI models with Azure's suite of security tools like Azure Security Center, Azure Active Directory, and Azure Key Vault.

This means you get fine-grained control for authorizing personnel at various levels within an organization through Role-Based Access Control (RBAC).

The service also adheres to global and industry-specific standards, including ISO 27001, HIPAA, and FedRAMP.

Here are some of the key security features that set Azure OpenAI apart: