Terraform setup to collect container logs in analytics workspace

Question

Terraform setup to collect container logs in analytics workspace

Erik Heeren 60

Hello,

I've got a small PoC setup involving:

An AKS cluster
A Log Analytics workspace
A data collection rule linking the AKS cluster to the log analytics workspace

When I deploy a demo namespace (the contoso pet shop) the application works, but none of my container logs show up in the analytics workspace. Once I set up logging through the portal logs start showing up. I'v made sure to verify that my terraform code matches the resources that get generated when I do it the clickops way.

What does the portal do in the background that my terraform code should be reproducing?

Here's the relevant parts of the terraform code:

resource "azurerm_resource_group" "poc_rg" {
  name     = var.resource_group_name
  location = var.aks_cluster_location
}

resource "azurerm_kubernetes_cluster" "poc_aks" {
  name                = var.aks_cluster_name
  location            = azurerm_resource_group.poc_rg.location
  resource_group_name = azurerm_resource_group.poc_rg.name
  dns_prefix          = var.aks_cluster_dnsprefix
  default_node_pool {
    name       = "default"
    node_count = var.aks_cluster_node_pool_count
    vm_size    = var.aks_cluster_node_pool_vmsize
  }
  identity {
    type = "SystemAssigned"
  }
  azure_active_directory_role_based_access_control {
    admin_group_object_ids = var.aks_cluster_admin_groups
  }
}

resource "azurerm_log_analytics_workspace" "aks_logs" {
  name                = "aks-logs"
  location            = var.location
  resource_group_name = azurerm_resource_group.poc_rg.name
  sku                 = "PerGB2018"
  retention_in_days   = 30
}

resource "azurerm_monitor_data_collection_rule" "aks_logs" {
  depends_on          = [azurerm_kubernetes_cluster.poc_aks]
  kind                = "Linux"
  location            = var.location
  name                = "aks-logs-collection-rule"
  resource_group_name = azurerm_resource_group.poc_rg.name
  data_flow {
    destinations = ["ciworkspace"]
    streams      = ["Microsoft-ContainerLog", "Microsoft-ContainerLogV2", "Microsoft-KubeEvents", "Microsoft-KubePodInventory"]
  }
  data_sources {
    extension {
      extension_json = jsonencode({
        dataCollectionSettings = {
          enableContainerLogV2   = true
          interval               = "1m"
          namespaceFilteringMode = "Off"
        }
      })
      extension_name = "ContainerInsights"
      name           = "ContainerInsightsExtension"
      streams        = ["Microsoft-ContainerLog", "Microsoft-ContainerLogV2", "Microsoft-KubeEvents", "Microsoft-KubePodInventory"]
    }
  }
  destinations {
    log_analytics {
      name                  = "ciworkspace"
      workspace_resource_id = azurerm_log_analytics_workspace.aks_logs.id
    }
  }
}

resource "azurerm_monitor_data_collection_rule_association" "aks_logs" {
  name                    = "collect-kubernetes-logs"
  target_resource_id      = azurerm_kubernetes_cluster.poc_aks.id
  data_collection_rule_id = azurerm_monitor_data_collection_rule.aks_logs.id
}

One extra annoying detail: if I delete all resources in my subscription and then redeploy everything, I still get to see logs from the previous deployment. Does "delete" mean something else on azure than it does in the rest of the world?

Bharath Y P 2,485 Reputation points Microsoft External Staff Moderator

2025-11-28T10:38:53.6766667+00:00

Hello Erik Heeren, I just wanted to kindly follow up to check if you had a chance to review my earlier response. I hope the information shared was helpful. Please feel free to reach out if you have any additional questions or need further clarification. If the answer addressed your query, it would be great if you could accept it and up-vote. This not only helps us but also makes it easier for other community members who might face the same issue. Thanks
Erik Heeren 60 Reputation points

2025-11-28T14:50:30.58+00:00
Hi @Bharath Y P ,

Thanks for your comprehensive answer! Some follow-up questions:

I notice that when I deploy a completely new namespace / application in a cluster that already has a different namespace in which I manually enabled monitoring, logs from this new namespace will automatically start appearing in the analytics workspace. Is this expected? Will all namespaces after the first manually enabled one behave like this or is it random?

By the crucial link that is the data collection rule association, you mean this, right?
resource "azurerm_monitor_data_collection_rule_association" "aks_logs" { name = "collect-kubernetes-logs" target_resource_id = azurerm_kubernetes_cluster.poc_aks.id data_collection_rule_id = azurerm_monitor_data_collection_rule.aks_logs.id
So I guess we can consider that part as being okay?

As you can see from the terraform the only identity I configure is this: identity { type = "SystemAssigned" } Is that enough? Where in the portal can I find which identity is assigned to the cluster?

I'm not sure what you mean by "the diagnostic settings" - can you elaborate?

About the deletion issue: I have found the recycle bin for log analytics workspaces, but the only option I have there is to recover them. Unlike for deleted key vaults, there is no purge button. I did find that the azurerm provider supports a log_analytics_workspace block in which you can set permanently_delete_on_destroy to true, so that takes care of that.

I'm currently looking into applying the Enable Azure Monitor for VMs with Azure Monitoring Agent(AMA) policy set to my resource group. Unfortunately it reports that all of my resources are compliant when clearly the AKS nodes are not. What am I missing here to make sure the policy set installs AMA on the AKS nodes? Is this even a possible solution or am I going in completely the wrong direction?

(note: I get the same result when I use the Enable Azure Monitor for VMSS with Azure Monitoring Agent(AMA) builtin policy set: all resources reported to be compliant)

Terraform code:

data "azurerm_policy_set_definition" "install_monitoring_agent" { name = "924bfe3a-762f-40e7-86dd-5c8b95eb09e6" } resource "azurerm_resource_group_policy_assignment" "aks-logging" { name = "InstallMonitoringAgent" resource_group_id = azurerm_resource_group.poc_rg.id policy_definition_id = data.azurerm_policy_set_definition.install_monitoring_agent.id enforce = true location = var.location identity { type = "SystemAssigned" } parameters = <<PARAMS { "bringYourOwnUserAssignedManagedIdentity": {"value": false}, "dcrResourceId": {"value": "${azurerm_monitor_data_collection_rule.aks_logs.id}"} } PARAMS } resource "azurerm_resource_group_policy_remediation" "aks-logging" { name = "remediateinstallmonitoringagent" resource_group_id = azurerm_resource_group.poc_rg.id policy_assignment_id = azurerm_resource_group_policy_assignment.aks-logging.id }
Erik Heeren 60 Reputation points

2025-11-28T16:10:30.43+00:00
Update: the policy set was not the way to go. Turns out I was missing this in the azurerm_kubernetes_cluster resource:

oms_agent { log_analytics_workspace_id = azurerm_log_analytics_workspace.aks_logs.id msi_auth_for_monitoring_enabled = false }

I'm still not getting ContainerLogV2, but at least I'm already getting ContainerLog and that's good enough for now.
Bharath Y P 2,485 Reputation points Microsoft External Staff Moderator

2025-12-02T01:17:40.21+00:00
Hello Erik Heeren, thanks for the update. below is the brief clarification to your question:

Yes, this is expected behavior. Once the Azure Monitor Agent (AMA) extension is installed and a Data Collection Rule Association (DCRA) is linked to the AKS cluster, the agent collects logs cluster-wide. The AMA runs as a DaemonSet on every node, scraping container stdout/stderr across all namespaces. That’s why when you deploy a new namespace/application, its logs automatically appear in the Log Analytics workspace. It’s not random all namespaces will behave this way unless you explicitly filter in the DCR.

As per your code, it tells the AMA agent which DCR to follow. So yes, you can consider that part okay, provided the AMA extension is also deployed.

A system-assigned identity is usually sufficient. The AMA uses this identity to authenticate and send data to the Log Analytics workspace.

Diagnostic settings are Azure Monitor configurations attached to the AKS resource itself. They’re separate from container logs collected by AMA. They capture control-plane logs and metrics, You can send them to Log Analytics, Storage, or Event Hub. When you enable monitoring, the portal automatically creates these diagnostic settings. For more details refer to this document Diagnostic settings in Azure Monitor - Azure Monitor | Microsoft Learn

The built‑in Azure Policy initiatives for “Enable Azure Monitor for VMs with AMA” and “Enable AMA on VMSS” are explicitly scoped to Microsoft.Compute/virtualMachines and Microsoft.Compute/virtualMachineScaleSets resources. They are designed to install the Azure Monitor Agent on regular VMs or VMSS instances

AKS nodes, however, are not exposed as standalone VM resources in Azure Policy. They are created and managed inside a hidden “MC_…” resource group by the AKS resource provider, and Azure Policy does not treat them as first‑class VM objects. Because of this:

The policy definitions never evaluate AKS nodes.

The initiative finds no non‑compliant VM/VMSS resources.

The compliance state is reported as “compliant,” even though AMA is not installed on AKS nodes

Use Azure Policy to Install the Azure Monitor Agent - Azure Monitor | Microsoft Learn

As per the terraform code you shared, that policy set won’t ever install AMA on AKS nodes. It targets VMs/VMSS (Microsoft.Compute), not AKS managed clusters. That’s why your assignment + remediation show “compliant” even though the cluster isn’t emitting logs the policy never evaluates AKS.

Enable AMA for VMs/VMSS) applies to VM resources, not AKS. Passing a dcrResourceId does not make it AKS‑aware; the policy’s resource selectors won’t touch the AKS managed cluster or its nodes. The remediation job only tries to deploy AMA to VM/VMSS types present in that RG. AKS nodes are hidden behind the AKS RP, so the policy sees no non‑compliant resources and reports “all compliant.”

You need the AKS extension, DCE, DCR and DCRA stack (container insights v2). Use Terraform resources that target the AKS managed cluster.

1 answer

Your answer

Bharath Y P 2,485 Reputation points Microsoft External Staff Moderator

2025-11-28T10:38:53.6766667+00:00

Hello Erik Heeren, I just wanted to kindly follow up to check if you had a chance to review my earlier response. I hope the information shared was helpful. Please feel free to reach out if you have any additional questions or need further clarification. If the answer addressed your query, it would be great if you could accept it and up-vote. This not only helps us but also makes it easier for other community members who might face the same issue. Thanks
Erik Heeren 60 Reputation points

2025-11-28T16:10:30.43+00:00

Update: the policy set was not the way to go. Turns out I was missing this in the azurerm_kubernetes_cluster resource:

oms_agent { log_analytics_workspace_id = azurerm_log_analytics_workspace.aks_logs.id msi_auth_for_monitoring_enabled = false }

I'm still not getting ContainerLogV2, but at least I'm already getting ContainerLog and that's good enough for now.

Answer 1

Hello Erik Heeren,

We understand that you configured an Azure AKS Cluster, Log Analytics Workspace (LAW), and Data Collection Rule (DCR) using Terraform. The application logs were not appearing in the LAW until the user manually clicked the "enable monitoring" option in the Azure portal.

The primary reason for the failure was the lack of the Data Collection Rule Association (DCRA) and the missing Azure Monitor Agent (AMA) extension deployment.

In the Azure portal's "enable monitoring" button performs several implicit configuration steps that your Terraform code must define explicitly. While you've likely created the DCR and the Log Analytics Workspace (LAW), the vital steps to enable data flow are usually the Agent Deployment and the DCR Association.

Logs won't flow until the Azure Monitor Agent (AMA) is running on your cluster nodes. The portal's "enable monitoring" feature installs the Container Insights extension (Microsoft.AzureMonitor.Containers), which deploys the AMA as a DaemonSet to collect and send logs/metrics.
The DCR defines what data to collect and where to send it. However, the cluster needs an explicit instruction telling its onboarded AMA agent which DCR it should follow. This crucial link is the Data Collection Rule Association (DCRA).
The AMA agent uses a Managed Identity to authenticate and send data. This identity requires permission to write to the Log Analytics workspace. While the oms_agent configuration often handles this implicitly, an explicit grant ensures no permission failures.

Could you please help us with the below details:

Have you verified that the managed identity for your AKS has been granted necessary permissions (like Log Analytics Contributor) to the Log Analytics workspace?
Can you confirm whether the diagnostic settings in your Terraform match the settings that get created when you configure them via the portal?

When you delete AKS, DCRs, and even the cluster, the workspace (or its tables) remains unless you explicitly delete it. Container logs (ContainerLogV2) are stored in the workspace, not in the cluster. By default, Log Analytics keeps data for 30 days (or longer if you configured retention). So even if the cluster is gone, the logs remain until retention expires or you purge them. Removing associations stops new ingestion, but historical data stays in the workspace. Certain Azure resources (e.g., storage accounts, workspaces) have a “soft delete” or delayed purge for recovery purposes. Deleting the resource doesn’t instantly erase its data.

Share via

Terraform setup to collect container logs in analytics workspace

1 answer

Your answer