Tutorial: Connect to Azure OpenAI Service in AKS using a connection string (preview)

Article
09/11/2024

In this tutorial, you learn how to create a pod in an Azure Kubernetes (AKS) cluster, which talks to Azure OpenAI Service using a connection string. You complete the following tasks:

Create an AKS cluster and Azure OpenAI Service with gpt-4 model deployment.
Create a connection between the AKS cluster and Azure OpenAI with Service Connector.
Clone a sample application that will talk to the OpenAI service from an AKS cluster.
Deploy the application to a pod in the AKS cluster and test the connection.
Clean up resources.

Warning

Microsoft recommends that you use the most secure authentication flow available. The authentication flow described in this procedure requires a very high degree of trust in the application, and carries risks that are not present in other flows. You should only use this flow when other more secure flows, such as managed identities, aren't viable. See the tutorial using a managed identity.

Prerequisites

An Azure account with an active subscription. Create an account for free.
- Use the Bash environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.
- If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.
  - If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.
  - When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use extensions with the Azure CLI.
  - Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.
Docker and kubectl to manage container images and Kubernetes resources.
A basic understanding of containers and AKS. Get started from preparing an application for AKS.
Access permissions to create Azure OpenAI Service resources and deploy models.

Create Azure resources

Create a resource group for this tutorial.

az group create \
    --name MyResourceGroup \
    --location eastus

Create an AKS cluster with the following command, or by referring to the AKS quickstart. This cluster is where we create the service connection and pod definition and deploy the sample application.
```
az aks create \
    --resource-group MyResourceGroup \
    --name MyAKSCluster \
    --enable-managed-identity \
    --node-count 1 \
   --generate-ssh-keys
```

Connect to the cluster using the az aks get-credentials command.

az aks get-credentials \
    --resource-group MyResourceGroup \
    --name MyAKSCluster

Create an Azure OpenAI Service resource using the az cognitiveservices account create command. Optionally refer to this tutorial for more instructions. Azure OpenAI Service is the target service that we'll connect to the AKS cluster.

az cognitiveservices account create \
    --resource-group MyResourceGroup \
    --name MyOpenAIService \
    --location eastus \
    --kind OpenAI \
    --sku s0 \
    --custom-domain myopenaiservice \
    --subscription <SubscriptionID>

Deploy a model with the az cognitiveservices deployment create command. The model is used in the sample application to test the connection.

az cognitiveservices account deployment create \
    --resource-group MyResourceGroup \
    --name MyOpenAIService
    --deployment-name MyModel \
    --model-name gpt-4 \
    --model-version 0613 \
    --model-format OpenAI \
    --sku-name "Standard"
    --capacity 1

Create an Azure Container Registry (ACR) resource with the az acr create command, or referring to this tutorial. The registry hosts the container image of the sample application, which the AKS pod definition consumes.
```
az acr create \
    --resource-group MyResourceGroup \
    --name myregistry \
    --sku Standard
```
Enable anonymous pull using az acr update command so that the AKS cluster can consume the images in the registry.
```
az acr update \
    --resource-group MyResourceGroup \
    --name myregistry \
    --anonymous-pull-enabled
```

Create a service connection in AKS with Service Connector (preview)

Create a service connection between an AKS cluster and Azure OpenAI Service in the Azure portal or the Azure CLI.

Portal
Azure CLI

Refer to the AKS service connection quickstart for instructions to create a new connection and fill in the settings referring to the examples in the following table. Leave all other settings with their default values.

Basics tab:

Setting	Example value	Description
Kubernetes namespace	default	The Kubernetes namespace.
Service type	OpenAI Service	The target service type.
Connection name	openai_conn	Use the connection name provided by Service Connector or choose your own connection name.
Subscription	`<MySubscription>`	The subscription used for Azure OpenAI Service.
OpenAI	`<MyOpenAIService>`	The target Azure OpenAI service you want to connect to.
Client type	Python	The code language or framework you use to connect to the target service.

Authentication tab:

Authentication Setting Example value Description

Authentication type Connection String Service Connector authentication type.

Authentication Setting	Example value	Description
Authentication type	Connection String	Service Connector authentication type.

Once the connection has been created, you can view its details in the Service Connector pane.

Use the Azure CLI command to create a service connection to the Azure OpenAI service, providing the following information:

Source compute service resource group name: the resource group name of the AKS cluster.
AKS cluster name: the name of your AKS cluster that connects to the target service.
Target service resource group name: the resource group name of the Azure OpenAI service.
OpenAI service name: the Azure OpenAI service that is connected.

az aks connection create cognitiveservices --secret

Clone sample application

Clone the sample repository:

git clone https://github.com/Azure-Samples/serviceconnector-aks-samples.git

Go to the repository's sample folder for Azure OpenAI:

cd serviceconnector-aks-samples/azure-openai-connection-string

Replace the <MyModel> placeholder in the app.py file with the model name we deployed.

Build and push container images

Build and push the images to your container registry using the Azure CLI az acr build command.
```
az acr build --registry <MyRegistry> --image sc-demo-openai-connstr:latest ./
```
View the images in your container registry using the az acr repository list command.
```
az acr repository list --name <MyRegistry> --output table
```

Run application and test connection

Replace the placeholders in the pod.yaml file in the azure-openai-connection-string folder.
- Replace <YourContainerImage> with the name of the image we built earlier. For example: <MyRegistry>.azurecr.io/sc-demo-openai-connstr:latest.
- Replace <SecretCreatedByServiceConnector> with the secret created by Service Connector. You may check the secret name in the Azure portal, in the Service Connector pane.
Deploy the pod to your cluster with the kubectl apply command. Install kubectl locally using the az aks install-cli command if it isn't installed. The command creates a pod named sc-demo-openai-connstr in the default namespace of your AKS cluster.
```
kubectl apply -f pod.yaml
```
Check if the deployment is successful by viewing the pod with kubectl.
```
kubectl get pod/sc-demo-openai-connstr
```
Check that the connection is working by viewing the logs with kubectl.
```
kubectl logs pod/sc-demo-openai-connstr
```

Clean up resources

If you don't need these resources anymore, clean up the Azure resources created in this tutorial by deleting the resource group.

az group delete \
    --resource-group MyResourceGroup

Next steps

Read the following articles to learn more about Service Connector concepts and how it helps AKS connect to Azure services.

Learn about Service Connector concepts

Use Service Connector to connect an AKS cluster to other cloud services

Share via