Lakeflow Connect CDC Gateway stuck in WAITING_FOR_RESOURCES due to EDv4 QuotaExceeded (Classic Compute)

Hi @Naga Vardhan Sankar Raparthi ,

Hi Naga,

Good diagnosis — the failure is entirely on the gateway's classic compute, not Azure SQL or CDC configuration. But the node types aren't actually locked: they're fixed in the UI, and settable through a compute policy or the Pipelines API.

Why it requests EDv4

The documented gateway compute policy pins:

{
  "driver_node_type_id": { "type": "fixed", "value": "Standard_E64d_v4" },
  "node_type_id":        { "type": "fixed", "value": "Standard_F4s" }
}

Standard_E64d_v4 is EDv4. Since your quota there is 0, provisioning fails before anything else runs.

Fix: create a custom compute policy with a different driver family

Per the SQL Server connector requirements, the gateway accepts "unrestricted permissions to create clusters, or a custom policy (API only)." Create a Job Compute policy with the required overrides, plus your own node types:

{
  "cluster_type":   { "type": "fixed", "value": "dlt" },
  "num_workers":    { "type": "unlimited", "defaultValue": 1, "isOptional": true },
  "runtime_engine": { "type": "fixed", "value": "STANDARD", "hidden": true },
  "driver_node_type_id": { "type": "fixed", "value": "Standard_E8ds_v5" },
  "node_type_id":        { "type": "fixed", "value": "Standard_F4s_v2" }
}

Pick driver/worker SKUs from families where you already have quota. Two constraints from the docs:

• Minimum 8 cores total for efficient extraction from the source.
• Databricks recommends the smallest possible worker nodes — worker size doesn't affect gateway performance, so keep the workers small and put the capacity in the driver.

Then attach the policy to the gateway pipeline via the Pipelines API:

{
  "clusters": [
    {
      "label": "default",
      "policy_id": "<your-policy-id>",
      "apply_policy_default_values": true
    }
  ]
}

"apply_policy_default_values": true is required for the policy defaults to actually take effect.

In parallel — the quota route is still viable

Note your error says Additional Required: 4, not 64. That's a very small ask. If the portal's quota-increase option is greyed out for EDv4 in your region, raise an Azure support request under Service and subscription limits (quotas) → Compute-VM (cores-vCPUs) rather than using the self-service form. Request +4 (or +8) standardEDv4Family cores in your target region and reference this error text. Greyed-out self-service usually means the family needs manual approval in that region, not that it's unobtainable.

Before choosing a region, verify capacity

az vm list-skus --location centralindia --size Standard_E --all --output table
az vm list-skus --location centralindia --size Standard_F --all --output table

Look for rows where Restrictions = None. This avoids the pattern of moving workspaces between regions and hitting the same wall — quota being available isn't the same as the SKU being unrestricted.

Recommended order: custom compute policy first (it unblocks you today without waiting on Azure), quota ticket in parallel as a fallback.

References

• Ingest data from SQL Server — requirements and gateway compute policy
• Configure classic compute for pipelines
• Managed database connectors overview

One caution: don't manually stop a running gateway — it must run continuously so change logs aren't truncated at the source. Re-deploying with the new policy is fine at this stage since it never started.

Kind Regards,
Microsoft Support Team.

Hi @Naga Vardhan Sankar Raparthi

Based on the information you've shared, this appears to be a limitation of the pipeline-managed Classic Compute rather than an issue with your Azure SQL Database or workspace configuration.

Since the CDC Gateway is managed by Databricks, the VM family is selected automatically. If the service is still attempting to provision Standard EDv4 instances, and your subscription has an EDv4 core quota of 0 (with quota requests unavailable because the SKU is retired), the gateway won't be able to start and will remain in WAITING_FOR_RESOURCES.

A few things to confirm:

• Verify whether Serverless or Pro/Standard Compute is supported for your Lakeflow Connect CDC scenario, as these may avoid the legacy Classic Compute dependency.
• If Classic Compute is currently required, there is no customer-facing option to override the managed gateway's VM family.
• Since you've reproduced the issue across multiple regions and confirmed that other VM families have available quota, this points to the managed service requesting an unavailable SKU rather than a workspace configuration issue.

At this point, I'd recommend opening a Databricks Support case (or Azure Support if using Azure Databricks) and providing:

• Workspace ID and region
• Pipeline ID
• Cluster event logs showing the QuotaExceeded error
• Confirmation that the EDv4 quota is 0 and cannot be increased because the family is retired

They should be able to confirm whether this is a known issue with the current Lakeflow Connect CDC managed gateway provisioning logic or if there is a backend configuration or feature flag to use a supported VM family (such as EDv5/Esv5).

Please "Accept the Answer" if this information helped you. This will help us and others in the community as well.