Share via

Init scripts stored in volumes are not getting executed via Job Shared cluster via ADF

Ashwini Gaikwad 130 Reputation points
2024-11-18T11:49:41.7666667+00:00

Hello Team,

We have stored our init script in Azure databricks managed volumes path and we want to execute notebook via Job Shared cluster, the pipeline we are triggering is via ADF but the job cluster gives an error as

User's image

User's image

We are using Assigned Manged Identity as an authentication from adf to databricks and this identity has all the permissions on databricks resource and clusters, objects, also the adf service principal has all permissions.

Please let us know why is this issue occuring and how to resolve it. Since as per Microsoft article Shared Compute cluster, volumes are supported.User's image

Regards,

Ashwini G

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,240 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,923 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Smaran Thoomu 17,525 Reputation points Microsoft Vendor
    2024-11-19T16:08:02.0766667+00:00

    Hi @Ashwini Gaikwad
    Welcome to Microsoft Q&A platform and thanks for posting your query here.

    Based on our internal review, here are some suggestions to help resolve the issue:

    1. Log Examination: Please check the logs of the Job Shared cluster to identify any specific error messages related to the init script execution. This could offer more insights into the root cause of the issue.
    2. Allowlist Configuration for Shared Access Mode: For clusters operating in shared access mode, it is essential to ensure that init scripts are added to the allowlist. Without this, the scripts may not execute correctly. For guidance on how to configure the allowlist for libraries and init scripts on shared compute clusters, you can refer to this documentation.

    You can do something like below:

    User's image

    User's image

    I hope this information helps. If you have any further questions, please let me know.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.