Share via

Serverless compute plane networking

  • Article

This guide introduces tools to secure network access between the compute resources in the Azure Databricks serverless compute plane and customer resources. To learn more about the control plane and the serverless compute plane, see Azure Databricks architecture overview.

To learn more about classic compute and serverless compute, see Types of compute.

Important

Starting December 4, 2024, Databricks will begin charging for networking costs on serverless workloads that connect to external resources. Billing will be implemented gradually, and you might not be charged until after December 4, 2024. You won’t be charged retroactively for usage before billing is enabled. After billing is enabled, you may be charged for:

  • Private connectivity to your resources over Private Link. Data processing charges for private connectivity to your resources over Private Link are waived indefinitely. Per-hour charges will apply.
  • Public connectivity to your resources over NAT gateway.
  • Data transfer charges incurred, such as when serverless compute and the target resource are in different regions.

Serverless compute plane networking overview

Serverless compute resources run in the serverless compute plane, which is managed by Azure Databricks. Account admins can configure secure connectivity between the serverless compute plane and their resources. This network connection is labeled as 2 on the diagram below:

Network connectivity overview diagram

Connectivity between the control plane and the serverless compute plane is always over the cloud network backbone and not the public internet. For more information on configuring security features on the other network connections in the diagram, see Networking.

What is a network connectivity configuration (NCC)?

Serverless network connectivity is managed with network connectivity configurations (NCC). NCCs are account-level regional constructs that are used to manage private endpoints creation and firewall enablement at scale.

Account admins create NCCs in the account console and an NCC can be attached to one or more workspaces. An NCC enables firewalls and private endpoints:

  • Resource firewall enablement by subnets: An NCC enables Databricks-managed stable Azure service subnets for adding service endpoints to your resource firewalls for secure access to Azure resources from serverless SQL warehouses. When an NCC is attached to a workspace, serverless compute in that workspace uses one of those networks to connect the Azure resource using service endpoints. You can allow list those networks on your Azure resource firewall. The network rules are automatically added to the workspace storage account. See Configure a firewall for serverless compute access.
  • Private endpoints: When you add a private endpoint in an NCC, Azure Databricks creates a private endpoint request to your Azure resource. Once the request is accepted on the resource side, the private endpoint is used to access your Azure resource from the serverless compute plane. See Configure private connectivity from serverless compute.

NCC firewall enablement is supported from serverless SQL warehouses, jobs, notebooks, Delta Live Tables pipelines, and model serving CPU endpoints. NCC private endpoints are only supported from serverless SQL warehouses. They are not supported from other compute resources in the serverless compute plane.