An Azure machine learning service for building and deploying models.
Deploy the scikit-learn artifact to an Azure Machine Learning managed online endpoint with SDK v2; it provides a managed, scalable HTTPS/REST scoring URL for real-time inference.^1^
1. Package and register the model
Save the complete preprocessing-and-model pipeline where possible, so inference applies the same transformations as training.
import joblib
from sklearn.pipeline import Pipeline
# pipeline = Pipeline([...]) # preprocessing and estimator
# pipeline.fit(X_train, y_train)
joblib.dump(pipeline, "model/model.joblib")
Connect to the workspace and register the directory as a custom model:
from azure.ai.ml import MLClient
from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes
from azure.identity import DefaultAzureCredential
ml_client = MLClient(
DefaultAzureCredential(),
"<subscription-id>", "<resource-group>", "<workspace>"
)
model = ml_client.models.create_or_update(Model(
path="./model",
name="sklearn-production-model",
type=AssetTypes.CUSTOM_MODEL
))
Registration versions the artifact independently from the serving deployment. Azure Machine Learning recommends registering the model before deployment.^2^
2. Implement the scoring contract
Create src/score.py. Azure Machine Learning calls init() once when the container starts and run() for every request; loading the model in init() avoids reloading it for each prediction.^3^
import json
import joblib
import os
import numpy as np
def init():
global model
path = os.path.join(os.getenv("AZUREML_MODEL_DIR"), "model.joblib")
model = joblib.load(path)
def run(raw_data):
request = json.loads(raw_data)
rows = np.asarray(request["input_data"])
predictions = model.predict(rows)
return {"predictions": predictions.tolist()}
Use an input contract such as:
{
"input_data": [
[5.1, 3.5, 1.4, 0.2],
[6.2, 2.8, 4.8, 1.8]
]
}
Validate shape, types, missing values, and payload size inside run() and return non-sensitive error messages. Do not log secrets or unredacted personal data.
3. Create the endpoint and deployment
Define the runtime dependencies:
# conda.yml
name: sklearn-inference
channels:
- conda-forge
dependencies:
- python=3.10
- pip
- pip:
- scikit-learn
- joblib
- azureml-inference-server-http
Create the endpoint and deploy the registered model:
import uuid
from azure.ai.ml.entities import (
ManagedOnlineEndpoint, ManagedOnlineDeployment,
Environment, CodeConfiguration
)
endpoint_name = f"sklearn-api-{str(uuid.uuid4())[:8]}"
endpoint = ManagedOnlineEndpoint(name=endpoint_name, auth_mode="aad_token")
ml_client.online_endpoints.begin_create_or_update(endpoint).result()
deployment = ManagedOnlineDeployment(
name="blue", endpoint_name=endpoint_name, model=model,
environment=Environment(conda_file="conda.yml",
image="mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04"),
code_configuration=CodeConfiguration(code="./src", scoring_script="score.py"),
instance_type="Standard_DS3_v2", instance_count=1
)
ml_client.online_deployments.begin_create_or_update(deployment).result()
Assign production traffic after testing:
endpoint = ml_client.online_endpoints.get(endpoint_name)
endpoint.traffic = {"blue": 100}
ml_client.online_endpoints.begin_create_or_update(endpoint).result()
print(endpoint.scoring_uri)
For production managed endpoints, Microsoft recommends Microsoft Entra authentication (aad_token); it is identity-based and requires the caller to have scoring permission. Key authentication is simpler for development but uses non-expiring static keys.^4^
4. Test the REST API
The SDK handles endpoint authentication automatically:
response = ml_client.online_endpoints.invoke(
endpoint_name=endpoint_name,
request_file="./request.json"
)
print(response)
A generic REST client sends the JSON body to the scoring URI with a key or token in the Authorization header.^4^
curl --request POST "$SCORING_URI" \
--header "Authorization: Bearer $DATA_PLANE_TOKEN" \
--header "Content-Type: application/json" \
--data @request.json
5. Handle simultaneous requests
Use load testing to determine latency and throughput before changing concurrency:
- Increase
instance_countfor horizontal capacity. - Configure Azure Monitor autoscaling for variable traffic.
- Set
max_concurrent_requests_per_instanceonly to the concurrency the model can actually process. Its default is1; setting it above actual processing capacity can increase latency. - When using Azure Machine Learning Inference Server or inference images, set
WORKER_COUNTto enable corresponding worker concurrency.^5^ - Batch multiple rows into one request when that fits the application and model.
- Keep
run()stateless and avoid shared mutable state.
Autoscaling can use CPU metrics, schedules, or both. In Studio, open Endpoints → endpoint → Details → Configure auto scaling, choose Custom autoscale, and set minimum, maximum, default capacity, and scale rules.^6^
6. Monitor service and model quality
Treat these as separate monitoring layers:
- Service health: In Studio, open Assets → Endpoints → endpoint → View metrics. Track request latency, requests per minute, failures, CPU/GPU, memory, disk, and network usage. Enable diagnostic settings for
AmlOnlineEndpointTrafficLog,AmlOnlineEndpointConsoleLog, andAmlOnlineEndpointEventLog; enable Application Insights during deployment for live metrics, failures, transactions, and performance views.^7^ - Model quality: Enable production inference data collection, then configure Manage → Monitoring → Add. Azure Machine Learning monitoring supports data drift, prediction drift, data quality, feature-attribution drift, and model-performance signals with scheduled evaluation and alert thresholds.^8^
- Log model version and a correlation ID with each prediction. Join delayed ground truth to predictions and calculate business metrics such as accuracy, precision/recall, calibration, or error by segment.
7. Control production risk and cost
- Start with a CPU SKU appropriate for scikit-learn and right-size it from measured CPU, memory, and latency.
- Autoscale between a small minimum and a bounded maximum; use scheduled scaling for predictable peaks.
- Set workspace/subscription quotas and delete unused or failed deployments. Managed endpoint charges are primarily compute and networking.^9^
- Deploy a new model version as a separate
greendeployment under the same endpoint. Test it directly, mirror traffic if appropriate, shift a small live percentage, then move to 100%. This preserves the endpoint URL and provides rollback by changing traffic allocation.^10^
References
- Deploy and score a machine learning model by using an online endpoint
- Deploy a model as an online endpoint
- Deploy and score a machine learning model by using an online endpoint
- Authenticate clients for online endpoints
- CLI (v2) managed online deployment YAML schema
- Autoscale online endpoints in Azure Machine Learning
- Monitor online endpoints
- Azure Machine Learning model monitoring
- Manage and optimize Azure Machine Learning costs
- Perform safe rollout of new deployments for real-time inference