Use a custom container to deploy a model to an online endpoint

git clone https://github.com/Azure/azureml-examples --depth 1
cd azureml-examples/cli

git clone https://github.com/Azure/azureml-examples --depth 1
cd azureml-examples/cli

In the examples repository, most Python samples are under the sdk/python folder. For this article, go to the cli folder instead. The folder structure under the cli folder is slightly different than the sdk/python structure in this case. Most steps in this article require the cli structure.

To follow along with the example steps, see a Jupyter notebook in the examples repository. But in the following sections of that notebook, the steps run from the azureml-examples/sdk/python folder instead of the cli folder:

• 3. Test locally
• 5. Test the endpoint with sample data

Create YAML files for your endpoint and deployment

You can configure your cloud deployment by using YAML. For instance, to configure your endpoint, you can create a YAML file named tfserving-endpoint.yml that contains the following lines:

$schema: https://azuremlsdk2.blob.core.windows.net/latest/managedOnlineEndpoint.schema.json
name: tfserving-endpoint
auth_mode: aml_token

To configure your deployment, you can create a YAML file named tfserving-deployment.yml that contains the following lines:

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: tfserving-deployment
endpoint_name: tfserving-endpoint
model:
  name: tfserving-mounted
  version: <model-version>
  path: ./half_plus_two
environment_variables:
  MODEL_BASE_PATH: /var/azureml-app/azureml-models/tfserving-mounted/<model-version>
  MODEL_NAME: half_plus_two
environment:
  #name: tfserving
  #version: 1
  image: docker.io/tensorflow/serving:latest
  inference_config:
    liveness_route:
      port: 8501
      path: /v1/models/half_plus_two
    readiness_route:
      port: 8501
      path: /v1/models/half_plus_two
    scoring_route:
      port: 8501
      path: /v1/models/half_plus_two:predict
instance_type: Standard_DS3_v2
instance_count: 1

Connect to your Azure Machine Learning workspace

To configure your Azure Machine Learning workspace, take the following steps:

1. Import the required libraries:

   # Import the required libraries.
   from azure.ai.ml import MLClient
   from azure.ai.ml.entities import (
      ManagedOnlineEndpoint,
      ManagedOnlineDeployment,
      Model,
      Environment,
      CodeConfiguration,
   )
   from azure.identity import DefaultAzureCredential
   

2. Configure workspace settings and get a handle to the workspace:

   # Enter information about your Azure Machine Learning workspace.
   subscription_id = "<subscription-ID>"
   resource_group = "<resource-group-name>"
   workspace = "<Azure-Machine-Learning-workspace-name>"
   
   # Get a handle to the workspace.
   ml_client = MLClient(
     DefaultAzureCredential(), subscription_id, resource_group, workspace
   )
   

For more information, see Deploy and score a machine learning model by using an online endpoint.

Configure an online endpoint

Use the following code to configure an online endpoint. Keep the following points in mind:

• The name of the endpoint must be unique in its Azure region. Also, an endpoint name must start with a letter and only consist of alphanumeric characters and hyphens. For more information about naming rules, see Azure Machine Learning online endpoints and batch endpoints.
• For the auth_mode value, use key for key-based authentication. Use aml_token for Azure Machine Learning token-based authentication. A key doesn't expire, but a token does expire. For more information about authentication, see Authenticate clients for online endpoints.
• The description and tags are optional.

# To create a unique endpoint name, use a time stamp of the current date and time.
import datetime

online_endpoint_name = "endpoint-" + datetime.datetime.now().strftime("%m%d%H%M%f")

# Configure an online endpoint.
endpoint = ManagedOnlineEndpoint(
    name=online_endpoint_name,
    description="A sample online endpoint",
    auth_mode="key",
    tags={"env": "dev"},
)

Configure an online deployment

A deployment is a set of resources that are required for hosting the model that does the actual inferencing. You can use the ManagedOnlineDeployment class to configure a deployment for your endpoint. The constructor of that class uses the following parameters:

• name: The name of the deployment.
• endpoint_name: The name of the endpoint to create the deployment under.
• model: The model to use for the deployment. This value can be either a reference to an existing versioned model in the workspace or an inline model specification.
• environment: The environment to use for the deployment. This value can be either a reference to an existing versioned environment in the workspace or an inline environment specification.
• environment_variables: Environment variables that are set during deployment.
  • MODEL_BASE_PATH: The path to the parent folder that contains a folder for your model.
  • MODEL_NAME: The name of your model.
• instance_type: The virtual machine size to use for the deployment. For a list of supported sizes, see Managed online endpoints SKU list.
• instance_count: The number of instances to use for the deployment.

Use the following code to configure a deployment for your endpoint:

# create a blue deployment
model = Model(name="tfserving-mounted", version="1", path="half_plus_two")

env = Environment(
    image="docker.io/tensorflow/serving:latest",
    inference_config={
        "liveness_route": {"port": 8501, "path": "/v1/models/half_plus_two"},
        "readiness_route": {"port": 8501, "path": "/v1/models/half_plus_two"},
        "scoring_route": {"port": 8501, "path": "/v1/models/half_plus_two:predict"},
    },
)

blue_deployment = ManagedOnlineDeployment(
    name="blue",
    endpoint_name=online_endpoint_name,
    model=model,
    environment=env,
    environment_variables={
        "MODEL_BASE_PATH": "/var/azureml-app/azureml-models/tfserving-mounted/1",
        "MODEL_NAME": "half_plus_two",
    },
    instance_type="Standard_DS2_v2",
    instance_count=1,
)

Suppose your tfserving-deployment.yml file contains the following lines in its model section. In this section, the name value refers to the name that you use to register the model in Azure Machine Learning.

model:
    name: tfserving-mounted
    version: 1
    path: ./half_plus_two

Suppose you use the following code to create a Model class. In this code, the name value refers to the name that you use to register the model in Azure Machine Learning.

model = Model(name="tfserving-mounted", version="1", path="half_plus_two")

For example, you can add the model_mount_path parameter to your tfserving-deployment.yml file. You can also update the MODEL_BASE_PATH value in that file:

name: tfserving-deployment
endpoint_name: tfserving-endpoint
model:
  name: tfserving-mounted
  version: 1
  path: ./half_plus_two
model_mount_path: /var/tfserving-model-mount
environment_variables:
  MODEL_BASE_PATH: /var/tfserving-model-mount
...

For example, you can add the model_mount_path parameter to your ManagedOnlineDeployment class. You can also update the MODEL_BASE_PATH value in that code:

blue_deployment = ManagedOnlineDeployment(
    name="blue",
    endpoint_name=online_endpoint_name,
    model=model,
    environment=env,
    model_mount_path="/var/tfserving-model-mount",
    environment_variables={
        "MODEL_BASE_PATH": "/var/tfserving-model-mount",
    ...
)

After you construct your YAML file, use the following command to create your endpoint:

az ml online-endpoint create --name tfserving-endpoint -f endpoints/online/custom-container/tfserving/half-plus-two/tfserving-endpoint.yml

Use the following command to create your deployment. This step might run for a few minutes.

az ml online-deployment create --name tfserving-deployment -f endpoints/online/custom-container/tfserving/half-plus-two/tfserving-deployment.yml --all-traffic

Use the following code to create the endpoint in the workspace. This code uses the instance of MLClient that you created earlier. The begin_create_or_update method starts the endpoint creation. It then returns a confirmation response while the endpoint creation continues.

ml_client.begin_create_or_update(endpoint)

Create the deployment by running the following code:

ml_client.begin_create_or_update(blue_deployment)

RESPONSE=$(az ml online-endpoint invoke -n $ENDPOINT_NAME --request-file $BASE_PATH/sample_request.json)

Use the instance of MLClient that you created earlier to get a handle to the endpoint. Then use the invoke method and the following parameters to invoke the endpoint:

• endpoint_name: The name of the endpoint
• request_file: The file that contains the request data
• deployment_name: The name of the deployment to test in the endpoint

For the request data, you can use a sample JSON file from the example repository.

# Test the blue deployment by using some sample data.
response = ml_client.online_endpoints.invoke(
    endpoint_name=online_endpoint_name,
    deployment_name="blue",
    request_file="sample_request.json",
)

az ml online-endpoint delete --name tfserving-endpoint

ml_client.online_endpoints.begin_delete(name=online_endpoint_name)