Configure Azure Monitor pipeline

The Azure Monitor pipeline extends the data collection capabilities of Azure Monitor to edge and multicloud environments. This article provides details on enabling and configuring the Azure Monitor pipeline in your environment.

Configuration methods

Start with the prerequisites and cert-manager installation steps in this article. Then use one of the following articles depending on your preferred configuration method:

• Configure Azure Monitor pipeline using the Azure portal for a simplified configuration experience that abstracts away the individual components of the pipeline. This method is recommended if you are new to the pipeline or prefer a more guided experience.
• Configure Azure Monitor pipeline using CLI or ARM templates for more advanced configuration options such as enabling the cache and configuring custom tables. This method is recommended if you are comfortable with CLI or ARM templates and want more control over the configuration of your pipeline.

Prerequisites

• Arc-enabled Kubernetes cluster in your own environment with an external IP address. See Connect an existing Kubernetes cluster to Azure Arc for details on enabling Arc for a cluster.
• The Arc-enabled Kubernetes cluster must have the custom locations features enabled. See Create and manage custom locations on Azure Arc-enabled Kubernetes.
• Log Analytics workspace in Azure Monitor to receive the data from the pipeline. See Create a Log Analytics workspace in the Azure portal for details on creating a workspace.
• The following resource providers must be registered in your Azure subscription. See Azure resource providers and types.
  • Microsoft.Insights
  • Microsoft.Monitor
• Install cert-manager as described in the following section.

Install cert-manager for Arc-enabled Kubernetes

This section describes how to install cert-manager as an Azure Arc extension. Installing cert-manager is required for Azure Monitor pipeline.

Installing cert-manager as a cluster managed extension (CME) will register the cert-manager and trust-manager services on your cluster.

Remove any existing instances of cert‑manager and trust‑manager from the cluster. Any open source versions must be removed before installing the Microsoft version.

The specific steps for removal will depend on your installation method. See Uninstalling cert-manager and Uninstalling trust-manager for detailed guidance. If you used Helm for installation, use the following command to check which namespace cert-manager and trust-manager installed using this command.

helm list -A | grep -E 'trust-manager|cert-manager'

If you have an existing cert-manager extension installed, uninstall it using the following commands:

export RESOURCE_GROUP="<resource-group-name>"
export CLUSTER_NAME="<arc-enabled-cluster-name>"
export LOCATION="<arc-enabled-cluster-location"

NAME_OF_OLD_EXTENSION=$(az k8s-extension list --resource-group ${RESOURCE_GROUP} --cluster-name ${CLUSTER_NAME})
az k8s-extension delete --name ${NAME_OF_OLD_EXTENSION} --cluster-name ${CLUSTER_NAME} \
  --resource-group ${RESOURCE_GROUP} --cluster-type connectedClusters

Use the following command to connect your cluster to Arc if it wasn't already connected.

az connectedk8s connect --name ${CLUSTER_NAME} --resource-group ${RESOURCE_GROUP} --location ${LOCATION}

Install the cert‑manager extension using the following command:

az k8s-extension create \
  --resource-group ${RESOURCE_GROUP} \
  --cluster-name ${CLUSTER_NAME} \
  --cluster-type connectedClusters \
  --name "azure-cert-management" \
  --extension-type "microsoft.certmanagement" \
  --release-train stable

Verify configuration

Once you've complete the configuration using your chosen method, use the following steps verify that the pipeline is running correctly in your environment.

Verify pipeline components running in the cluster

In the Azure portal, navigate to the Kubernetes services menu and select your Arc-enabled Kubernetes cluster. Select Services and ingresses and ensure that you see the following services:

• <pipeline name>-external-service
• <pipeline name>-service

Click on the entry for <pipeline name>-external-service and note the IP address and port in the Endpoints column. This is the external IP address and port that your clients will send data to. See Retrieve ingress endpoint for retrieving this address from the client.

Verify heartbeat

Each pipeline configured in your pipeline instance will send a heartbeat record to the Heartbeat table in your Log Analytics workspace every minute. The contents of the OSMajorVersion column should match the name your pipeline instance. If there are multiple workspaces in the pipeline instance, then the first one configured will be used.

Retrieve the heartbeat records using a log query as in the following example:

Troubleshooting

kubectl get pods -n mon
NAME                                                              READY   STATUS             RESTARTS       AGE
edge-pipeline-pipeline-operator-controller-manager-6f847d4njwcn   1/2     CrashLoopBackOff   11 (24s ago)   31m

kubectl logs <operator-pod-name> -n mon

AttemptTlsBootstrap returned an error:  failed to apply resource: the server could not find the requested resource (patch clusterissuers.meta.k8s.io arc-amp-selfsigned-cluster-issuer)
Please ensure Azure Arc Cert Manager Extension is installed on the cluster.
panic: failed to apply resource: the server could not find the requested resource (patch clusterissuers.meta.k8s.io arc-amp-selfsigned-cluster-issuer)

az k8s-extension list --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --query "[?extensionType=='microsoft.certmanagement'].{Name:name, State:provisioningState}" -o table