Custom Catalog Not Visible to ADF and Looker in Databricks

Question

Custom Catalog Not Visible to ADF and Looker in Databricks

Adrian P - Studytube 25

Hi Databricks community,

I’ve created a catalog called azure_data_factory and a database called studytube_azure_mysql_production. Within that database, I have three tables: academies, academy_naming_settings, and users. See the left side of the screenshot.

However, when I try to connect Databricks to Azure Data Factory, ADF only displays the tables and the database studytube_azure_sql_production under the default catalog. See the right side of the screenshot.

I’ve already granted the necessary permissions (USE and SELECT).

I’m experiencing the same issue when connecting Databricks to Looker — Looker can only see the tables and the database under the default catalog.

Ideally, I want all data replicated from ADF to be placed in schemas under the azure_data_factory catalog. Has anyone encountered this issue?

Pilladi Padma Sai Manisha 495 Reputation points Microsoft External Staff Moderator

2025-11-27T05:15:07.2033333+00:00

Hi Adrian P - Studytube,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Adrian P - Studytube 25 Reputation points

2025-11-27T09:01:20.63+00:00

Hi @Pilladi Padma Sai Manisha ,

Thanks for answering my questions. The required permissions have been granted to the custom catalog, and I’ve waited several hours to see the effect (see the screenshot). Furthermore, both catalogs are using Unity Catalog (see the screenshot). Lastly, I’m running Databricks through a direct Microsoft Azure subscription, which should be compatible with Azure Data Factory.

However, ADF is still unable to detect the custom catalog, while Looker can see the contents inside it. I still suspect that the issue lies in the ADF configuration. Could you perhaps help me pinpoint it?

Regards,
Pilladi Padma Sai Manisha 495 Reputation points Microsoft External Staff Moderator

2025-11-27T15:23:39.6533333+00:00

Hi Adrian P - Studytube,

Thanks for the update! Since you’ve confirmed the necessary permissions are set, Unity Catalog is in use, and your Databricks environment is properly connected via Azure subscription, the fact that Looker can see the custom catalog but Azure Data Factory (ADF) can’t strongly suggests a configuration issue on the ADF side.

Here are some practical checks you can do in ADF:

Make sure the linked service in ADF explicitly specifies your custom catalog name in its connection settings. If this isn’t set, ADF will default to the default catalog.

When defining datasets or writing queries in ADF, always use fully qualified names including the catalog, e.g., azure_data_factory.studytube_azure_mysql_production.<table_name>. Without this, ADF might not recognize tables outside the default catalog.

Double-check the permissions and network access for the identity running your ADF integration runtime. It needs catalog-level permissions and proper connectivity to Databricks.

Confirm you’re using connection methods and drivers in ADF that fully support Unity Catalog. Older or generic drivers may limit what catalogs are visible.

Sometimes ADF caches metadata, so try refreshing or recreating the linked service or datasets to make sure ADF fetches the latest catalog information.

You can also try running a direct query from ADF using fully qualified catalog names or set up a Databricks SQL endpoint connected to your custom catalog for use within ADF. This approach often helps isolate if the problem is with metadata retrieval or connectivity.

Let me know if you want detailed walkthroughs on any of these steps. This should help pinpoint and resolve the configuration gap so ADF can see and work with your custom catalog just like Looker does.
Adrian P - Studytube 25 Reputation points

2025-11-27T15:54:35.2366667+00:00

Hi @Pilladi Padma Sai Manisha ,

Thanks for getting back to me. Let’s go through this one by one.

Make sure the linked service in ADF explicitly specifies your custom catalog name in its connection settings. If this isn’t set, ADF will default to the default catalog.

How can I configure that in ADF? Could you provide a screenshot showing this in ADF?
Pilladi Padma Sai Manisha 495 Reputation points Microsoft External Staff Moderator

2025-11-28T11:43:45.4466667+00:00
Hi Adrian P - Studytube,
The ADF Databricks Delta Lake connector doesn't have a catalog selector in the UI, so when you browse datasets, it only shows objects from the default catalog. That's why your azure_data_factory custom catalog isn't appearing in the ADF dropdown it's a UI limitation, not a Databricks configuration issue.

Your data is accessible. Even though the catalog doesn't show in the UI, you can read from it using fully qualified table names.

Use Query Mode with Fully Qualified Names

In your Copy Activity > Source:

Choose Use query instead of selecting a table

Enter your SQL with the full three-part table name (catalog.schema.table):

Plain Text

sqlSELECT * FROM azure_data_factory.studytube_azure_mysql_production.academies

For all three tables:

Plain Text

sqlSELECT * FROM azure_data_factory.studytube_azure_mysql_production.academiesUNION ALLSELECT * FROM azure_data_factory.studytube_azure_mysql_production.academy_naming_settingsUNION ALLSELECT * FROM azure_data_factory.studytube_azure_mysql_production.users

Execute the query ADF will correctly read from your custom catalog

Why this works: Query mode bypasses the UI dropdown limitation and goes directly to Databricks with the full catalog reference.

For Writing Data to Your Custom Catalog

If you need to write data back to your custom catalog tables, use an INSERT query:

Plain Text

sqlINSERT INTO azure_data_factory.studytube_azure_mysql_production.academiesSELECT * FROM [your source table]

This works the same way the fully qualified name tells Databricks which catalog to write to.

References:

Databricks - What are catalogs (3-level namespace)

Azure Databricks Names & Identifiers

Community - Delta Lake UI Limitation with UC

Databricks KB - Unity Catalog (Azure)
VRISHABHANATH PATIL 1,725 Reputation points Microsoft External Staff Moderator

2025-12-01T07:26:08.3266667+00:00

Hi @Adrian P - Studytube,

I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Adrian P - Studytube 25 Reputation points

2025-12-01T19:23:26.7366667+00:00

Hi @Pilladi Padma Sai Manisha and @VRISHABHANATH PATIL ,

Thank you for the guidance. I’ll have a look and get back to you if I have any follow-up questions.
Pilladi Padma Sai Manisha 495 Reputation points Microsoft External Staff Moderator

2025-12-02T21:11:58.9233333+00:00

Hi Adrian P - Studytube,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

2 answers

Your answer

Pilladi Padma Sai Manisha 495 Reputation points Microsoft External Staff Moderator

2025-11-27T05:15:07.2033333+00:00

Hi Adrian P - Studytube,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Adrian P - Studytube 25 Reputation points

2025-11-27T09:01:20.63+00:00

Hi @Pilladi Padma Sai Manisha ,

Thanks for answering my questions. The required permissions have been granted to the custom catalog, and I’ve waited several hours to see the effect (see the screenshot). Furthermore, both catalogs are using Unity Catalog (see the screenshot). Lastly, I’m running Databricks through a direct Microsoft Azure subscription, which should be compatible with Azure Data Factory.

However, ADF is still unable to detect the custom catalog, while Looker can see the contents inside it. I still suspect that the issue lies in the ADF configuration. Could you perhaps help me pinpoint it?

Regards,
Pilladi Padma Sai Manisha 495 Reputation points Microsoft External Staff Moderator

2025-11-27T15:23:39.6533333+00:00

Hi Adrian P - Studytube,

Thanks for the update! Since you’ve confirmed the necessary permissions are set, Unity Catalog is in use, and your Databricks environment is properly connected via Azure subscription, the fact that Looker can see the custom catalog but Azure Data Factory (ADF) can’t strongly suggests a configuration issue on the ADF side.

Here are some practical checks you can do in ADF:

Make sure the linked service in ADF explicitly specifies your custom catalog name in its connection settings. If this isn’t set, ADF will default to the default catalog.

When defining datasets or writing queries in ADF, always use fully qualified names including the catalog, e.g., azure_data_factory.studytube_azure_mysql_production.<table_name>. Without this, ADF might not recognize tables outside the default catalog.

Double-check the permissions and network access for the identity running your ADF integration runtime. It needs catalog-level permissions and proper connectivity to Databricks.

Confirm you’re using connection methods and drivers in ADF that fully support Unity Catalog. Older or generic drivers may limit what catalogs are visible.

Sometimes ADF caches metadata, so try refreshing or recreating the linked service or datasets to make sure ADF fetches the latest catalog information.

You can also try running a direct query from ADF using fully qualified catalog names or set up a Databricks SQL endpoint connected to your custom catalog for use within ADF. This approach often helps isolate if the problem is with metadata retrieval or connectivity.

Let me know if you want detailed walkthroughs on any of these steps. This should help pinpoint and resolve the configuration gap so ADF can see and work with your custom catalog just like Looker does.
Adrian P - Studytube 25 Reputation points

2025-11-27T15:54:35.2366667+00:00

Hi @Pilladi Padma Sai Manisha ,

Thanks for getting back to me. Let’s go through this one by one.

Make sure the linked service in ADF explicitly specifies your custom catalog name in its connection settings. If this isn’t set, ADF will default to the default catalog.

How can I configure that in ADF? Could you provide a screenshot showing this in ADF?
Pilladi Padma Sai Manisha 495 Reputation points Microsoft External Staff Moderator

2025-11-28T11:43:45.4466667+00:00

Hi Adrian P - Studytube,
The ADF Databricks Delta Lake connector doesn't have a catalog selector in the UI, so when you browse datasets, it only shows objects from the default catalog. That's why your azure_data_factory custom catalog isn't appearing in the ADF dropdown it's a UI limitation, not a Databricks configuration issue.

Your data is accessible. Even though the catalog doesn't show in the UI, you can read from it using fully qualified table names.

Use Query Mode with Fully Qualified Names

In your Copy Activity > Source:

Choose Use query instead of selecting a table

Enter your SQL with the full three-part table name (catalog.schema.table):

Plain Text

sqlSELECT * FROM azure_data_factory.studytube_azure_mysql_production.academies

For all three tables:

Plain Text

sqlSELECT * FROM azure_data_factory.studytube_azure_mysql_production.academiesUNION ALLSELECT * FROM azure_data_factory.studytube_azure_mysql_production.academy_naming_settingsUNION ALLSELECT * FROM azure_data_factory.studytube_azure_mysql_production.users

Execute the query ADF will correctly read from your custom catalog

Why this works: Query mode bypasses the UI dropdown limitation and goes directly to Databricks with the full catalog reference.

For Writing Data to Your Custom Catalog

If you need to write data back to your custom catalog tables, use an INSERT query:

Plain Text

sqlINSERT INTO azure_data_factory.studytube_azure_mysql_production.academiesSELECT * FROM [your source table]

This works the same way the fully qualified name tells Databricks which catalog to write to.

References:

Databricks - What are catalogs (3-level namespace)

Azure Databricks Names & Identifiers

Community - Delta Lake UI Limitation with UC

Databricks KB - Unity Catalog (Azure)
VRISHABHANATH PATIL 1,725 Reputation points Microsoft External Staff Moderator

2025-12-01T07:26:08.3266667+00:00

Hi @Adrian P - Studytube,

I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Adrian P - Studytube 25 Reputation points

2025-12-01T19:23:26.7366667+00:00

Hi @Pilladi Padma Sai Manisha and @VRISHABHANATH PATIL ,

Thank you for the guidance. I’ll have a look and get back to you if I have any follow-up questions.
Pilladi Padma Sai Manisha 495 Reputation points Microsoft External Staff Moderator

2025-12-02T21:11:58.9233333+00:00

Hi Adrian P - Studytube,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

Answer 1

It seems that you are experiencing visibility issues with your custom catalog azure_data_factory when connecting Databricks to Azure Data Factory (ADF) and Looker. Here are a few things you can check:

Permissions: Although you mentioned that you have granted the necessary permissions (USE and SELECT), ensure that these permissions are correctly applied to the azure_data_factory catalog and its associated database and tables. Sometimes, permissions may not propagate as expected, so double-check the settings.
Catalog Visibility: Make sure that the catalog is properly registered in the Unity Catalog metastore. If the catalog is not registered correctly, external services like ADF and Looker may not be able to see it.
Connection Configuration: Verify that the connection settings in ADF and Looker are correctly configured to access the azure_data_factory catalog. Sometimes, the connection might default to the default catalog if not explicitly set.
Databricks Runtime Version: Ensure that you are using a compatible Databricks runtime version that supports Unity Catalog features, as some older versions may have limitations.
Check for Errors: Look for any error messages or logs in ADF and Looker that might indicate why the catalog is not visible. This can provide clues on what might be going wrong.

If you have checked all of these and the issue persists, consider reaching out to Databricks support for further assistance as they can provide more specific insights based on your account configuration.

References:

Answer 2

Hi Adrian P - Studytube,

Thankyou for Reaching Microsoft Q&A!
It looks like the main reason you’re only seeing tables and databases under the default catalog in Azure Data Factory (ADF) and Looker when connecting to Databricks is due to how permissions, catalog registration, and connection settings are configured.

First, double-check that the USE and SELECT permissions are granted directly on your custom catalog (azure_data_factory), along with its databases and tables, for the service accounts that ADF and Looker use. Sometimes permission changes don’t automatically apply everywhere, so it’s important to review them closely.

Also, make sure the custom catalog is properly registered in the Unity Catalog metastore and enabled for external access. If it’s not, tools like ADF and Looker won’t be able to see it and will default to showing objects in the default catalog.

Another common cause is related to the connection setup itself. Check that in ADF and Looker, the connection explicitly points to your custom catalog instead of defaulting to the default catalog. This often requires specifying the catalog name in the connection configuration.

Finally, confirm you’re running a Databricks runtime version that fully supports Unity Catalog features, especially external access. Older versions sometimes have limitations that affect catalog visibility.

By verifying these areas permissions on the catalog, proper catalog registration and external access, explicit connection configuration, and compatible runtime you should be able to get ADF and Looker to see and work with your custom catalog and its schemas instead of only the default catalog. This is a known pattern experienced by others integrating Databricks with external services. Hope this helps! Feel free to reach out if you need further assistance!

Share via

Custom Catalog Not Visible to ADF and Looker in Databricks

2 answers

Your answer