Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
This article explains how to use tags to attribute compute usage to specific workspaces, teams, projects, or users to support cost tracking and budgeting.
There are two types of tags:
- Default tags: Automatically applied by Databricks to cloud-deployed resources. These provide basic metadata like vendor, cluster ID, and creator.
- Custom tags: User-defined tags that you can add to compute resources and serverless workloads. These allow for granular tracking, reporting, and budgeting.
Warning
Tag data may be replicated globally. Do not use tag names or values that could compromise the security of your resources. For example, do not use tag names that contain personal or sensitive information.
Default tags
Azure Databricks automatically adds default tags to compute resources it deploys in your cloud account. These tags attribute the usage to Databricks and provide basic information about the resource, such as its name, ID, and creator.
The default tags automatically propagate to detailed cost analysis reports that you can access in the Azure portal.
Here is a cost analysis invoice details report in the Azure portal that details cost by clusterid tag over a one-month period:
Default tag keys and values
Azure Databricks adds the following default tags to compute resources:
| Tag key | Value |
|---|---|
Vendor |
Constant value: Databricks |
ClusterId |
Azure Databricks internal ID of the cluster |
ClusterName |
Name of the cluster |
Creator |
Username (email address) of the user who created the cluster |
RunName |
Job name (only propagates on jobs compute) |
JobId |
Job ID (only propagates on jobs compute) |
Compute used by Lakehouse monitoring includes these additional tags:
| Tag key | Value |
|---|---|
LakehouseMonitoring |
true |
LakehouseMonitoringTableId |
ID of the monitored table |
LakehouseMonitoringWorkspaceId |
ID of the workspace where the monitor was created |
LakehouseMonitoringMetastoreId |
ID of the metastore where the monitored table exists |
Azure Databricks adds the following default tags to pools and the compute resources created by pools.
| Tag key | Value |
|---|---|
Vendor |
Constant value: Databricks |
DatabricksInstancePoolCreatorId |
Azure Databricks internal ID of the user who created the pool |
DatabricksInstancePoolId |
Azure Databricks internal ID of the pool |
Custom tags
Custom tags let you attribute compute usage to specific teams, projects, or cost centers with more granularity than default tags. These tags are applied by users or admins and propagate to both your account's usage logs and applicable cloud resources. These tags are also used to create and monitor budgets in your Azure Databricks account.
Supported resources for custom tags
| Object | Tagging interface (UI) | Tagging interface (API) |
|---|---|---|
| Workspace | Azure Portal | Azure Resources API |
| Pool | Pools UI in the Azure Databricks workspace | Instance Pool API |
| All-purpose and job compute | Compute UI in the Azure Databricks workspace | Clusters API |
| SQL warehouse | SQL warehouse UI in the Azure Databricks workspace | Warehouses API |
Custom tags appear in lowercase in GCE logs. Characters are removed if they are not letters, numbers, underscores or dashes. For example, My Key becomes mykey and My.Val becomes myval.
Warning
Do not assign a custom tag with the key Name to a cluster. Every cluster has a tag Name whose value is set by Azure Databricks. If you change the value associated with the key Name, the cluster can no longer be tracked by Azure Databricks. As a consequence, the cluster might not be terminated after becoming idle and will continue to incur usage costs.
Tag serverless compute workloads
Important
This feature is in Public Preview.
To attribute serverless compute usage to users, groups, or projects, you can use serverless budget policies. When a user is assigned a serverless budget policy, their serverless usage is automatically tagged with their policy's custom tags. Serverless budget policies can be applied to serverless notebooks, jobs, pipelines, and model serving endpoints.
Note
Serverless compute usage is logged in your account's billable usage system table. The legacy DBU usage reports do not include serverless usage or serverless budget policy tags.
See Attribute usage with serverless budget policies.
Tag propagation
Workspace, pool, and cluster tags are aggregated by Azure Databricks and propagated to Azure VMs for cost analysis reporting. But pool and cluster tags are propagated differently from each other.
Workspace and pool tags are aggregated and assigned as resource tags of the Azure VMs that host the pools.
Workspace and cluster tags are aggregated and assigned as resource tags of the Azure VMs that host the clusters.
When clusters are created from pools, only workspace tags and pool tags are propagated to the VMs. Cluster tags are not propagated, in order to preserve pool cluster startup performance.
Tag conflict resolution
If a custom cluster tag, pool tag, or workspace tag has the same name as an Azure Databricks default cluster or pool tag, the custom tag is prefixed with an x_ when it is propagated.
For example, if a workspace is tagged with vendor = Azure Databricks, that tag will conflict with the default cluster tag vendor = Databricks. The tags will therefore be propagated as x_vendor = Azure Databricks and vendor = Databricks.
Warning
Conflicting custom tags added through compute policies do not resolve, causing the cluster or pool to fail with an invalid settings error. Ensure your compute policies do not add any conflicting tag names.
Tag enforcement
To enforce the use of specific custom tags, you can use compute policies. See Custom tag enforcement. To enforce custom tags on serverless compute workloads, use serverless budget policies.
Limitations
- It can take up to one hour for custom workspace tags to propagate to Azure Databricks after any change.
- No more than 50 tags can be assigned to an Azure resource. If the overall count of aggregated tags exceeds this limit,
x_-prefixed tags are evaluated in alphabetical order and those that exceed the limit are ignored. If allx_-prefixed tags are ignored and the count is till over the limit, the remaining tags are evaluated in alphabetical order and those that exceed the limit are ignored. - Tag keys and values can only contain letters, spaces, numbers, or the characters
+,-,=,.,_,:,/,@. Tags containing other characters are invalid. - If you change tag key names or values, these changes apply only after cluster restart or pool expansion.
- If the cluster's custom tags conflict with a pool's custom tags, the cluster can't be created.
- Newly added workspace tags do not automatically propagate to existing compute resources. To get new tags to propagate, open the compute resource's details page, click Edit, and then Confirm and restart.