Enable trace and collect feedback for a flow deployment (preview)

After deploying a generative AI app in production, app developers seek to enhance their understanding and optimize performance. Trace data for each request, aggregated metrics, and user feedback play critical roles.

In this article, you learn how to enable trace, collect aggregated metrics, and gather user feedback during inference time of your flow deployment.

Prerequisites

• The Azure CLI and the Azure Machine Learning extension to the Azure CLI. For more information, see Install, set up, and use the CLI (v2).
• An Azure Machine Learning workspace. If you don't have one, use the steps in the Quickstart: Create workspace resources article to create one.
• An Application Insights. Usually a machine learning workspace has a default linked Application Insights. If you want to use a new one, you can create an Application Insights resource.
• Python 3.9 or later, if you plan to use the Python code samples in this article.
• Learn how to build and test a flow in the prompt flow.
• Have a basic understanding of managed online endpoints. Managed online endpoints work with powerful CPU and GPU machines in Azure in a scalable, fully managed way that frees you from the overhead of setting up and managing the underlying deployment infrastructure. For more information on managed online endpoints, see Online endpoints and deployments for real-time inference.
• Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the owner or contributor role for the Azure Machine Learning workspace, or a custom role allowing "Microsoft.MachineLearningServices/workspaces/onlineEndpoints/". If you use studio to create and manage online endpoints and deployments, you need another permission "Microsoft.Resources/deployments/write" from the resource group owner. For more information, see Manage access to an Azure Machine Learning workspace.

Deploy a flow for real-time inference

After you test your flow properly, either a flex flow or a DAG flow, you can deploy the flow in production. In this article, use deploy a flow to Azure Machine Learning managed online endpoints as an example. For flex flows, you need to prepare the flow.flex.yaml file instead of flow.dag.yaml.

You can also deploy to other platforms, such as Docker container, Kubernetes cluster, and more.

Enable trace and collect system metrics for your deployment

If you use the studio UI to deploy, turn on Application Insights diagnostics in Advanced settings > Deployment step in the deploy wizard. This setting collects tracing data and system metrics to the workspace linked Application Insights.

If you use SDK or CLI, add the property app_insights_enabled: true in the deployment YAML file to collect data to the workspace linked Application Insights. You can also specify another Application Insights by using the environment variable APPLICATIONINSIGHTS_CONNECTION_STRING in the deployment YAML file as shown in the following example. You can find the connection string of your Application Insights in the Overview page in Azure portal.

# below is the property in deployment yaml
# app_insights_enabled: true

# you can also use the environment variable
environment_variables:
  APPLICATIONINSIGHTS_CONNECTION_STRING: <connection_string>

View tracing data in Application Insights

Traces record specific events or the state of an application during execution. They can include data about function calls, variable values, system events, and more. Traces help break down an application's components into discrete inputs and outputs, which is crucial for debugging and understanding an application. To learn more, see OpenTelemetry traces. The trace data follows OpenTelemetry specification.

You can view the detailed trace in the specified Application Insights. The following screenshot shows an example of an event of a deployed flow containing multiple nodes. In Application Insights, select Search under the Investigate category in the resource menu. Select each node to view its detailed trace.

The Dependency type events record calls from your deployments. The name of that event is the name of the flow folder. To learn more, see Transaction search and diagnostics in Application Insights.

View system metrics in Application Insights

You can find the workspace default Application Insights in your workspace overview page in Azure portal.

Open Application Insights, and select Usage and estimated cost from the left navigation. Under Send custom metrics to Azure Metric Store, select With dimensions, and save the change.

Select Metrics tab in the left navigation. Select promptflow standard metrics from the Metric Namespace, and you can explore the metrics from the Metric dropdown list with different aggregation methods.

Collect feedback and send to Application Insights

Prompt flow serving provides a new /feedback API to help you collect feedback. The feedback payload can be any JSON format data. PF serving just helps you save the feedback data to a trace span. The data is saved to the trace exporter target you configure. It also supports OpenTelemetry standard trace context propagation, so it respects the trace context set in the request header and uses that context as the request parent span context. You can leverage the distributed tracing functionality to correlate the feedback trace to its chat request trace.

The following sample code shows how to score a flow deployed managed endpoint enabled tracing and send the feedback to the same trace span of scoring request. The flow has inputs question and chat_history, and output answer. After scoring the endpoint, you collect feedback and send it to Application Insights specified when deploying the flow. You need to fill in the api_key value or modify the code according to your use case.

import urllib.request
import json
import os
from opentelemetry import trace, context
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.sdk.trace import TracerProvider

# Initialize your tracer
tracer = trace.get_tracer("my.genai.tracer")
trace.set_tracer_provider(TracerProvider())

# Request data goes here
# The example below assumes JSON formatting which may be updated
# depending on the format your endpoint expects.
# More information can be found here:
# https://learn.microsoft.com/azure/machine-learning/how-to-deploy-advanced-entry-script
data = {
    "question": "hello",
    "chat_history": []
}

body = str.encode(json.dumps(data))

url = 'https://basic-chat-endpoint.eastus.inference.ml.azure.com/score'
feedback_url = 'https://basic-chat-endpoint.eastus.inference.ml.azure.com/feedback'
# Replace this with the primary/secondary key, AMLToken, or Microsoft Entra ID token for the endpoint
api_key = ''
if not api_key:
    raise Exception("A key should be provided to invoke the endpoint")

# The azureml-model-deployment header will force the request to go to a specific deployment.
# Remove this header to have the request observe the endpoint traffic rules
headers = {'Content-Type':'application/json', 'Authorization':('Bearer '+ api_key), 'azureml-model-deployment': 'basic-chat-deployment' }

try:
    with tracer.start_as_current_span('genai-request') as span:

        ctx = context.get_current()
        TraceContextTextMapPropagator().inject(headers, ctx)
        print(headers)
        print(ctx)
        req = urllib.request.Request(url, body, headers)
        response = urllib.request.urlopen(req)

        result = response.read()
        print(result)

        # Now you can process the answer and collect feedback
        feedback = "thumbdown"  # Example feedback (modify as needed)

        # Make another request to save the feedback
        feedback_body = str.encode(json.dumps(feedback))
        feedback_req = urllib.request.Request(feedback_url, feedback_body, headers)
        urllib.request.urlopen(feedback_req)


except urllib.error.HTTPError as error:
    print("The request failed with status code: " + str(error.code))

    # Print the headers - they include the requert ID and the timestamp, which are useful for debugging the failure
    print(error.info())
    print(error.read().decode("utf8", 'ignore'))

You can view the trace of the request along with feedback in Application Insights.

Advanced usage: Export trace to custom OpenTelemetry collector service

In some cases, you might want to export the trace data to your deployed OTel collector service. Enable this export by setting OTEL_EXPORTER_OTLP_ENDPOINT. Use this exporter when you want to customize your own span processing logic and your own trace persistent target.

Next steps

• Migrate prompt flow to Microsoft Agent Framework
• Troubleshoot errors of managed online endpoints