Spark config through Databricks

Mxxx 0

Hi, I am trying to access storage account through databricks (thru notebook) with storage access key.

I keep encountering error when I do spark.config.set

spark.conf.set(
        f"fs.azure.account.key.{storage_account_name}.blob.core.windows.net",
        access_key
    )

I tried many alternative code with spark.conf.set, but I keep encountering the same error message

Configuration fs.azure.account.key.{storage_account_name}.blob.core.windows.net is not available. SQLSTATE: 42K0I

May I know what could be the issue?
I am using serverless compute.

Thank you.

1 answer

Deepanshukatara-6769 11,000 Reputation points

2024-11-20T07:01:10.53+00:00
Hello , Welcome to MS Q&A

To access a storage account through Databricks using a storage access key, you can set Spark properties in your Databricks notebook or cluster configuration. Here’s how to do it:

spark.conf.set( "fs.azure.account.key.<storage-account>.dfs.core.windows.net", dbutils.secrets.get(scope="<scope>", key="<storage-account-access-key>") )

Once configured, you can interact with resources in the storage account using URIs. It is recommended to use the abfss driver for enhanced security.

References:

Connect to Azure Data Lake Storage Gen2 and Blob Storage

Kindly accept answer if it works

Please let us know if any questions

Thanks

Deepanshu
Please sign in to rate this answer.
Mxxx 0 Reputation points

2024-11-20T07:16:05.0233333+00:00

Hello
I tried the below

spark.conf.set( "fs.azure.account.key.<storage-account>.dfs.core.windows.net", dbutils.secrets.get(scope="<scope>", key="<storage-account-access-key>") )

or even (Since we using blob and not dfs)

spark.conf.set( "fs.azure.account.key.<storage-account>.blob.core.windows.net", dbutils.secrets.get(scope="<scope>", key="<storage-account-access-key>") )

We are still encountering the same error
Configuration fs.azure.account.key.<storage-account>.blob/dfs.core.windows.net is not available. SQLSTATE: 42K0I

Deepanshukatara-6769 11,000 Reputation points

2024-11-20T07:22:11.5766667+00:00

Ok can you please confirm that are you replacing below points?

<storage-account> with the Azure Storage account name.

<scope> with the Azure Databricks secret scope name.

<storage-account-access-key> with the name of the key containing the Azure storage account access key.

Mxxx 0 Reputation points

2024-11-20T07:25:12.95+00:00

Yes, I did.
I didn't put the value here for confidentiality

Mxxx 0 Reputation points

2024-11-20T07:30:41.2033333+00:00

Yes, I did.
I didn't update here for confidentiality.

Deepanshukatara-6769 11,000 Reputation points

2024-11-20T08:41:47.6266667+00:00

Ok , It seems that the configuration for fs.azure.account.key.<storage-account>.blob/dfs.core.windows.net is not available, which could indicate that the storage account is either not configured correctly or that the necessary keys are not set up in the cluster configuration.

To resolve this, you may need to check the following:

Ensure that the storage account is correctly set up and that you have the correct access keys.

If you have recently changed the access key for the storage account, you will need to update the cached copy in the cluster's core-site.xml configuration.

You can remove the storage account and then add it back to refresh the configuration.

If you have secured your storage account with firewall settings, ensure that the necessary exceptions are enabled for HDInsight to access the storage account.

References:

Add additional storage accounts to HDInsight

Known issues

If still issue occurs , lets have a short session and connect over call so that we can deep dive in to this

Mxxx 0 Reputation points

2024-11-21T03:20:48.82+00:00

I have verified my access key by doing local CLI and it works, and the rest is not applicable.
For my storage account network, It is set as 'Enabled from all networks' , thus there isn't any special firewall setting.

Will love to have a short session to troubleshoot this issue together.
Let me know how can we start a call on this. Thank you!

Mxxx 0 Reputation points

2024-11-22T08:32:46.41+00:00

Hi, just to give some update. It seems like this is due to using 'serverless compute'. The same code used on personal compute can be run successfully. Thank you.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Spark config through Databricks

1 answer

Your answer