ADF Azure Databricks Linked Service job cluster using policyId that specifies pool id

The issue you're encountering arises because the ADF databricks linked service requires proper alignment between the policyId and the cluster configuration parameters. Here's how you can configure the linked service correctly to use a job policy that specifies the driver and worker pool IDs without encountering the errors:

Steps to Configure the Linked Service

Set the policyId Only (No newClusterNodeType or newClusterDriverNodeType) A job policy in Databricks defines the allowed configurations for a job cluster. When you specify a policyId, the policy enforces configurations such as driver and worker node types, instance pool IDs, and other cluster settings. As a result, specifying additional properties like node_type_id alongside policyId causes conflicts.

In your case:

• Remove newClusterNodeType and newClusterDriverNodeType entirely from the typeProperties JSON.
  • Ensure that the policy associated with the policyId includes instance pool settings for both the driver and worker nodes.

1. Dynamic JSON Example Here's how your typeProperties section in the linked service should look when using a policyId:

      {
          "type": "AzureDatabricks",
          "typeProperties": {
              "domain": "https://<databricks-instance>.azuredatabricks.net",
              "accessToken": "<your-databricks-access-token>",
              "policyId": "<your-policy-id>",
              "newClusterVersion": "<runtime-version>", // Specify the Databricks runtime version
              "newClusterNumOfWorker": "<number-of-workers>"
          }
      }
      
      
   

2. Ensure Policy Compliance Verify with your Databricks administrator that:
   • The specified policyId includes instance pool settings for the driver and worker nodes.
   • The policy allows for flexibility in setting the number of workers and runtime version, as these may need to be specified dynamically in your ADF pipeline.
3. Permissions Check If you do not have visibility into instance pools for security reasons, you can still use the policyId to enforce instance pool usage. However, ensure the policy includes these settings:
   • "instancePoolId" for both driver and worker nodes.
   • Any constraints that restrict or specify acceptable configurations.
4. Test Configuration After updating the linked service, test the configuration by triggering a minimal pipeline that uses the linked service. If the error persists, ask your Databricks admin to verify the compatibility of the job policy with the ADF pipeline requirement