Implementing Azure Openai Private Link for Enterprise

Azure OpenAI Private Link allows you to connect to OpenAI services over a private endpoint, reducing the attack surface of your Azure resources.

To set up Azure OpenAI Private Link, you need to create a private endpoint in the Azure portal, which will be used to connect to the OpenAI service.

A private endpoint is a network interface that connects your Azure resources to a private network, providing a secure and isolated connection to the OpenAI service.

The private endpoint will be created in the same region as your OpenAI service, ensuring low latency and high performance.

To configure Azure OpenAI Private Link, you'll need to create a shared private link between your search resource and Azure OpenAI resource. This is only applicable for S2 pricing tier search resources, as it requires private endpoint support for indexers with a skill set.

You'll need to select Resource type as Microsoft.CognitiveServices/accounts and Group ID as openai_account in the search documentation. This will change step 8 of the data ingestion architecture diagram from bypass trusted service to shared private link.

To allow access to your Storage Account from Azure OpenAI and Azure AI Search, you'll need to set up Storage Account to bypass your Azure OpenAI and Azure AI Search as trusted services based on managed identity. This involves navigating to your storage account networking tab, choosing "Selected networks", and then selecting Allow Azure services on the trusted services list to access this storage account and clicking Save.

To configure your resources for optimal secure usage, create a resource group to organize all relevant resources. This includes Azure OpenAI resources, Azure AI search resources, and storage accounts.

First, you'll need to create a resource group. This will help you keep all your resources in one place, making it easier to manage and secure them.

The resources in the resource group include, but are not limited to, Azure OpenAI resources, Azure AI search resources, and storage accounts. You can also use the resource group to manage other Azure resources.

To set up managed identities for your resources, see the management API reference documentation. This will help you secure your resources and ensure that only authorized services can access them.

By following these steps, you'll be able to configure your resources for optimal secure usage. This will help protect your resources from unauthorized access and ensure that they are only accessed by authorized services.

To establish a dedicated connection to Azure OpenAI, you'll need to create two critical connections: one between the model serving endpoint and your AWS VPC, and another from your VPC to Azure OpenAI.

Databricks' serverless compute plane networking is managed by Network Connectivity Configuration (NCC), which offers two options for connecting to Azure OpenAI.

Each NCC container currently provides two options for connecting to Azure OpenAI, so you can choose the one that best suits your needs.

To eliminate public access entirely, you can construct a VPN connection between Amazon VPC and Azure VNet, but you'll need to reach out to your account team for guidance on how to do this.

To establish a secure connection between your Azure OpenAI and Azure AI Search, you need to set up Azure OpenAI to bypass Azure AI Search as a trusted service based on managed identity.

Azure OpenAI identifies the traffic from your Azure AI Search by verifying the claims in the JSON Web Token (JWT). Azure AI Search must use the system assigned managed identity authentication to call the custom skill web API.

You can skip this step if you have a shared private link for your Azure AI Search resource.

To lock down access to the internet from your serverless workloads on AWS, but enable access to Azure Open AI, you can use a dedicated, per-customer connection.

This connection can be established by deploying HAProxy on EC2 instances as a Layer 4 forwarding mechanism.

Requests from the model serving nodes are routed through PrivateLink to HAProxy servers, which then forward these requests directly to Azure Open AI.

To enhance the enterprise-readiness of the solution, several features can be implemented:

Azure Open AI Firewall can also be used to restrict access to only authorized IP addresses for greater security.

You can establish an Elastic IP (EIP) pool for all HAProxy servers and configure Azure Open AI service to permit access solely from this IP pool.

This can be done by establishing an existing NCC object and a Network Policy (with restricted access) object that you wish to use for your workspace.

Creating a private endpoint is a crucial step in establishing a secure and dedicated connection to Azure OpenAI. To create a private endpoint, you'll need to select the NCC you created and navigate to Private endpoint rules, where you can click "Add private endpoint rule" to create a new rule.

The Endpoint service is the service name of the VPC endpoint service that you created, while the Domain names is the FQDN of the destination resource, such as dais24-aoai-demo.openai.azure.com. You can also use Terraform to create a Private Endpoint and deploy it into the backend subnet of your Azure Virtual Network.

Once the private endpoint rule is created, it will show a PENDING status, but after refreshing the page, it will show an ESTABLISHED status. This indicates that the private endpoint is now established and ready for use.

To create a private endpoint with Terraform, you'll need to create a Private Endpoint and deploy it into the backend subnet of your Azure Virtual Network.

First, you'll need to create a private DNS zone for privatelink.openai.azure.com with an A record that resolves your individual service URL to the private IP address of your Private Endpoint.

Here's a step-by-step process to achieve this:

Create a Private Endpoint

1. Create a Private Endpoint and deploy it into the backend subnet of your Azure Virtual Network.

2. Create a private DNS zone for privatelink.openai.azure.com with an A record that resolves your individual service URL to the private IP address of your Private Endpoint.

Configure Azure OpenAI Firewall

1. Restrict access to only authorized IP addresses for greater security.

2. Establish an Elastic IP (EIP) pool for all HAProxy servers and configure Azure OpenAI service to permit access solely from this IP pool.

Note: You can skip this step if there is an existing NCC object and a Network Policy (with restricted access) object that you wish to use for your workspace.

Creating a shared private link is only applicable for the S2 pricing tier, specifically for search resources that require private endpoint support for indexers with a skill set.

You should skip this step if you're using a basic or standard pricing tier, or if it's your first time setting up resources securely.

To create a shared private link, you'll need to follow the search documentation and select Resource type as Microsoft.CognitiveServices/accounts and Group ID as openai_account.

This will allow you to connect your search resource to your Azure OpenAI resource securely.

To verify private network access to the search service, you need to test connections. This involves opening PowerShell in the Remote Desktop of your VM and entering a command to perform a name server lookup.

The command is nslookup [search service name].search.windows.net, which will return a message similar to the one shown in Example 1, including the private link and public endpoint of your search service.

To connect privately to the search service, you'll need to create an index using the REST API, which requires the search service endpoint and admin API-key. This is a crucial step in confirming that your service is fully operational.

Here are the steps to test your private endpoint:

To establish a dedicated connection, you'll need to create two critical connections: one between the model serving endpoint and your customer's VPC in AWS, and another from your VPC to Azure OpenAI.

Databricks' serverless compute plane networking is managed by Network Connectivity Configuration (NCC), which offers two options for connection.

Each NCC container currently has two options for networking. You can choose one that suits your needs.

Constructing a VPN connection between Amazon VPC and Azure VNet is a viable option if you want to eliminate public access entirely. Please reach out to your account team for guidance on this route.

To test connections, you'll need to verify private network access to the search service and connect privately using the Private Endpoint.

Open PowerShell in the Remote Desktop of your VM and enter nslookup [search service name].search.windows.net. You'll receive a message similar to this: Server: UnKnown Address: 168.63.129.16 Non-authoritative answer: Name: [search service name].privatelink.search.windows.net Address: 10.0.0.5 Aliases: [search service name].search.windows.net

From the VM, connect to the search service and create an index. You can follow the quickstart to create a new search index in your service using the REST API. Setting up requests from a Web API test tool requires the search service endpoint (https://[search service name].search.windows.net) and the admin api-key you copied in a previous step.

To confirm that the service is fully operational, complete the quickstart from the VM. Closing the remote desktop connection to myVM is the next step.

To verify that your service isn't accessible on a public endpoint, open a REST client on your local workstation and attempt the first several tasks in the quickstart. If you receive an error that the remote server doesn't exist, you successfully configured a private endpoint for your search service.

With Azure OpenAI Private Link, you can customize and manage your connections to OpenAI models in a secure and scalable way.

You can create multiple private endpoints to connect to different OpenAI models or services, each with its own unique DNS name.

Each private endpoint can be used to connect to a specific OpenAI model or service, allowing you to manage and isolate traffic to each one.

You can also use Azure OpenAI Private Link to manage access to your OpenAI models and services, including controlling who can access them and what actions they can perform.

Azure OpenAI Private Link supports multiple authentication methods, including Azure Active Directory (Azure AD) and Azure Key Vault.

To lock down access to Azure OpenAI, you can restrict internet access from your serverless workloads on AWS, while enabling access to Azure OpenAI through a dedicated connection.

By using an Azure Network Security Group, you can create two security rules: prevent-all and allow-apps-to-openai-https. The prevent-all rule turns off all TCP inbound traffic for the backend subnet with destination port 443, while the allow-apps-to-openai-https rule allows inbound TCP traffic from the apps subnet to the IP address of the Private Endpoint with destination port 443.

To associate the Network Security Group with your backend subnet, you can use an azurerm_subnet_network_security_group_association resource.

To enable trusted service, Azure OpenAI needs to bypass Azure AI Search as a trusted service based on managed identity. This is done by setting networkAcls.bypass as AzureServices from the management API.

Azure OpenAI identifies the traffic from Azure AI Search by verifying the claims in the JSON Web Token (JWT). Azure AI Search must use the system assigned managed identity authentication to call the custom skill web API.

Here's a summary of the steps to enable trusted service:

Azure OpenAI Private Link is a secure and dedicated connection to Azure OpenAI, allowing you to focus on data and AI use-cases without worrying about unauthorized access. This is achieved through Databricks serverless networking, which provides a simple and secure way to connect to Azure OpenAI.

One of the key features of Azure OpenAI Private Link is the ability to lock down access to the internet from your serverless workloads on AWS, while still enabling access to Azure OpenAI through a dedicated, per-customer connection.

To set up a cross-cloud scenario, you can use HAProxy on EC2 instances as a Layer 4 forwarding mechanism. This allows you to route requests from your model serving nodes through PrivateLink to HAProxy servers, which then forward these requests directly to Azure OpenAI.

Here are some key features of the setup:

To test connections, you can verify private network access to the search service and connect privately to the service using the Private Endpoint. You can do this by following the steps outlined in the Azure documentation.

In Azure, you can create a new search service with a private endpoint by following these steps: