Azure Batch + Azure Monitor Agent: correlating Computer names with pool names (Python SDK)

Francesco Cipolla 60 Reputation points
2026-04-28T09:10:02.7733333+00:00

Introduction

We're using the Python azure-mgmt-batch SDK to create Azure Batch pools backed by VMSS, with Azure Monitor Agent (AMA) enabled via the extensions field in the ARM pool definition. AMA is shipping Linux performance counters (CPU, memory etc.) to a Log Analytics workspace via a Data Collection Rule.

Here is the relevant pool creation code just for reference:

 pool_params = Pool(
      identity=BatchPoolIdentity(
          type=PoolIdentityType.USER_ASSIGNED,
          user_assigned_identities={
              node_monitoring_identity_resource_id: UserAssignedIdentities()
          },
      ),
      vm_size=self.pool_vm_size,
      deployment_configuration=DeploymentConfiguration(
          virtual_machine_configuration=VirtualMachineConfiguration(
              image_reference=ImageReference(...),
              node_agent_sku_id=sku_to_use,
              extensions=[
                  VMExtension(
                      name="AzureMonitorAgent",
                      publisher="Microsoft.Azure.Monitor",
                      type="AzureMonitorLinuxAgent",
                      type_handler_version="1.0",
                      auto_upgrade_minor_version=True,
                      enable_automatic_upgrade=True,
                      settings={
                          "authentication": {
                              "managedIdentity": {
                                  "identifier-name": "mi_res_id",
                                  "identifier-value": node_monitoring_identity_resource_id,
                              }
                          }
                      },
                  )
              ],
          )
      ),
      ...
  )
  batch_mgmt_client.pool.create(
      resource_group_name=batch_account_resource_group,
      account_name=batch_account_name,
      pool_name=pool_name,
      parameters=pool_params,
  )

  # DCR association created on the pool ARM resource after creation
  monitor_client.data_collection_rule_associations.create(
      resource_uri=created_pool.id,
      association_name="ama-dcr-association",
      body=DataCollectionRuleAssociationProxyOnlyResource(
          data_collection_rule_id=dcr_resource_id,
      ),
  )

The problem:

Querying CPU metrics per node works fine, example KQL query:

Perf
  | where ObjectName == "Processor" and CounterName == "% Processor Time" and InstanceName == "total"
  | summarize avg(CounterValue) by Computer, bin(TimeGenerated, 5m)

Result:

┌─────────────────────────┬──────────────────────┬──────────────────┐
│        Computer         │    TimeGenerated     │ avg_CounterValue │
├─────────────────────────┼──────────────────────┼──────────────────┤
│ <computer_name_a>000000 │ 2026-04-27T23:30:00Z │ 78.4             │ ├─────────────────────────┼──────────────────────┼──────────────────┤
│ <computer_name_a>000001 │ 2026-04-27T23:30:00Z │ 81.2             │
├─────────────────────────┼──────────────────────┼──────────────────┤
│ <computer_name_a>000002 │ 2026-04-27T23:30:00Z │ 75.9             │
├─────────────────────────┼──────────────────────┼──────────────────┤
│ <computer_name_b>000000 │ 2026-04-27T23:30:00Z │ 12.1             │
├─────────────────────────┼──────────────────────┼──────────────────┤
│ <computer_name_b>000001 │ 2026-04-27T23:30:00Z │ 10.8             │
└─────────────────────────┴──────────────────────┴──────────────────┘

I need to be able to correlate these Computer names with the Batch pool name in order to aggregate metrics per pool in Grafana, so that the end user doens't need to double check the computer names. This is especially important for easily querying performance metrics of pools which allocate dozens of nodes that are automatically deleted as soon as the corresponding batch job ends.

There is no field in the Perf table, the _ResourceId, or the Computer name that contains the Batch pool name.
The VMSS UUID and the Computer name prefix are both auto-generated by Azure and have no documented relationship to the pool name.

My initial goal was to inject a custom BatchPoolRegistry_CL table into the same Log Analytics workspace containing the PoolName → Computer mapping, so we can join it with the Perf table.
What I've investigated:

  • BatchManagementClient.pool.get() — ARM pool response does not expose the underlying VMSS resource group or UUID
  • compute_node.list() (legacy azure-batch SDK) — node IDs follow tvmps_{hex}_p format with no obvious link to the VMSS UUID or Computer name

Questions:

  1. Is there any API in azure-mgmt-batch (Python management SDK) that exposes the underlying VMSS resource group or a stable identifier correlating with what AMA reports as Computer?
  2. Alternatively, is there an officially supported way to associate a Batch pool name with its AMA-monitored nodes?

Thanks!

Azure Batch
Azure Batch

An Azure service that provides cloud-scale job scheduling and compute management.


Answer accepted by question author

Manish Deshpande 7,790 Reputation points Microsoft External Staff Moderator
2026-04-28T13:15:26.5066667+00:00

Yes you are understanding the recommendation correctly, and yes — you will still need a custom table for historical correlation.

  • VMSS-based lookups are only valid while the VMSS exists
  • Once a Batch pool is deleted (and its VMSS is torn down), Log Analytics retains Perf data but loses the control‑plane context
  • Therefore, the correct and supported approach is:
    1. Use resourceTags on the Batch pool to discover the VMSS while it exists
    2. Extract VMSS name → Pool name
    3. Persist that mapping into a custom Log Analytics table
    4. Join historical Perf._ResourceId against that table

You did not miss any important steps.

This behavior is by design and not specific to AMA.

  • Azure Batch deliberately abstracts away the underlying infrastructure
  • The Batch control plane never exposes VMSS identity via azure-mgmt-batch
  • Azure Monitor (AMA) reports telemetry at the VM / VMSS layer, not the Batch layer
  • Once a VMSS is deleted:
    • Azure Resource Graph
    • ARM
    • Compute APIs no longer contain metadata
  • Log Analytics keeps Perf rows, but ResourceId becomes the only remaining anchor

There is currently no native field in Perf that stores the Batch pool name, nor is there a supported way to backfill that automatically after deletion.

Tag the Batch pool (control plane)

When creating the pool, set resourceTags:

pool_params = Pool(
    ...
    resource_tags={
        "BatchPoolName": pool_name,
        "BatchAccountName": batch_account_name
    },
    ...
)

Discover VMSS while it exists

Query Compute / ARM to find VMSS with that tag:

Tags["BatchPoolName"] == "<pool-name>"

https://learn.microsoft.com/en-us/rest/api/compute/virtual-machine-scale-sets/list?view=rest-compute-2025-11-01&tabs=HTTP

Query historical Perf data reliably

Because _ResourceId is retained forever in Perf:

let PoolMap = BatchPoolRegistry_CL
    | project PoolName, VmssResourceIdPrefix;
Perf
| where ObjectName == "Processor"
| where CounterName == "% Processor Time"
| extend VmssPrefix = extract(@"virtualMachineScaleSets/([^/]+)", 1, _ResourceId)
| join kind=leftouter PoolMap on $left.VmssPrefix == $right.VmssName
| summarize avg(CounterValue) by PoolName, bin(TimeGenerated, 5m)

Your understanding is exactly correct. The VMSS-based correlation works only while the backing infrastructure exists. For historical analysis, the supported pattern is to materialize the VMSS → Batch pool association at pool creation time and persist it into a custom Log Analytics table, then join against the _ResourceId column in Perf.

Thanks,
Manish.

Was this answer helpful?

1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
  1. AI answer

    2026-04-28T09:10:20.8266667+00:00
    1. The provided information about Azure Batch pools, nodes, and VM configuration does not describe any API in the azure-mgmt-batch Python management SDK that exposes the underlying VM scale set resource group, VMSS ID, or any other stable identifier that can be directly correlated with the Computer field reported by Azure Monitor Agent in the Perf table.
    2. The available documentation only covers:
      • How to define a pool with VirtualMachineConfiguration, including image_reference, node_agent_sku_id, and optional extensions.
      • General concepts of pools and nodes, including that Batch uses virtual machine scale sets under the hood and that each node has a unique name and IP address.
      • How to connect to Linux nodes via SSH and create users.
      • How to create pools from custom images or Azure Compute Gallery images.
      • Best practices for pool configuration and monitoring Batch with Azure Monitor.

    None of these describe an officially supported mechanism to:

    • Retrieve the backing VMSS resource or its identifiers from the Batch management plane, or
    • Join Batch pool metadata (such as pool name) with AMA-reported Computer values in Log Analytics.

    Within the provided material, there is therefore no documented, supported way to associate Batch pool names with AMA-monitored nodes via a stable identifier that appears in the Perf table.


    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.