AZF Dataflow error connection CosmosDB

Question

AZF Dataflow error connection CosmosDB

Erica de Graaf-Borkent 0

User's image

I have created a linked service and the testconnection is succeeded

But when I use this linked service in a dataflow and test the connection I get the following error:
User's image

DetailsStatus code 400, "{"error":{"code":"InvalidSubscriptionId","message":"The provided subscription identifier 'resourceGroups' is malformed or invalid."}}"

Does anyone know a solution for this problem?

Smaran Thoomu 22,505 Reputation points Microsoft External Staff

2025-04-22T12:17:29.8066667+00:00
Hi @Erica de Graaf-Borkent
Based on the error message:

Status code 400: {"code":"InvalidSubscriptionId","message":"The provided subscription identifier 'resourceGroups' is malformed or invalid."}

It looks like the issue is happening within the ADF Data Flow source setup, not in the linked service directly (which passed the connection test successfully).

Even though your linked service to Cosmos DB is working fine, the Data Flow connector expects a subscription ID and resource group to be correctly set, either directly or inferred, when configuring the source dataset in the Data Flow.

I would suggest following the below recommendations:

Review the dataset used in the Data Flow

Double-check that the dataset used in the Data Flow has the correct linked service selected.

Make sure that you're not using resourceGroups (which seems to be incorrectly used as a subscription ID in the background).

If you're manually entering account URIs, ensure there are no typos or leftover values from earlier setups.

Try switching “Account selection method”

If you're using "Enter manually", try switching to "From Azure subscription" (if possible). This might help ADF internally pick the correct subscription/resourceGroup values.

Publish All and Revalidate

After correcting the dataset and Data Flow setup, hit "Publish All" and try validating the pipeline again.

Let us know if that helps or if you would like us to review the dataset configuration further. Happy to assist.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Smaran Thoomu 22,505 Reputation points Microsoft External Staff

2025-04-23T07:56:35.8333333+00:00

@Erica de Graaf-Borkent We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dileep Raj Narayan Thumula 0 Reputation points Microsoft External Staff

2025-04-24T11:47:49.7033333+00:00

Hello @Erica de Graaf-Borkent
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Erica de Graaf-Borkent 0 Reputation points

2025-04-25T13:02:15.2+00:00

I think my linked service is correctly configured, because when I use the Data flow debug mode my Data flow shows all steps correctly, with a preview of the data from the CosmosDB I expected to see.
I didn't use the system assigned identity in a POC I did before with the same CosmosDB, I used the account key to configure the linked service. And in that case my pipeline did run successfully.

1 answer

Your answer

Smaran Thoomu 22,505 Reputation points Microsoft External Staff

2025-04-22T12:17:29.8066667+00:00

Hi @Erica de Graaf-Borkent
Based on the error message:

Status code 400: {"code":"InvalidSubscriptionId","message":"The provided subscription identifier 'resourceGroups' is malformed or invalid."}

It looks like the issue is happening within the ADF Data Flow source setup, not in the linked service directly (which passed the connection test successfully).

Even though your linked service to Cosmos DB is working fine, the Data Flow connector expects a subscription ID and resource group to be correctly set, either directly or inferred, when configuring the source dataset in the Data Flow.

I would suggest following the below recommendations:

Review the dataset used in the Data Flow

Double-check that the dataset used in the Data Flow has the correct linked service selected.

Make sure that you're not using resourceGroups (which seems to be incorrectly used as a subscription ID in the background).

If you're manually entering account URIs, ensure there are no typos or leftover values from earlier setups.

Try switching “Account selection method”

If you're using "Enter manually", try switching to "From Azure subscription" (if possible). This might help ADF internally pick the correct subscription/resourceGroup values.

Publish All and Revalidate

After correcting the dataset and Data Flow setup, hit "Publish All" and try validating the pipeline again.

Let us know if that helps or if you would like us to review the dataset configuration further. Happy to assist.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Smaran Thoomu 22,505 Reputation points Microsoft External Staff

2025-04-23T07:56:35.8333333+00:00

@Erica de Graaf-Borkent We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dileep Raj Narayan Thumula 0 Reputation points Microsoft External Staff

2025-04-24T11:47:49.7033333+00:00

Hello @Erica de Graaf-Borkent
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Erica de Graaf-Borkent 0 Reputation points

2025-04-25T13:02:15.2+00:00

I think my linked service is correctly configured, because when I use the Data flow debug mode my Data flow shows all steps correctly, with a preview of the data from the CosmosDB I expected to see.
I didn't use the system assigned identity in a POC I did before with the same CosmosDB, I used the account key to configure the linked service. And in that case my pipeline did run successfully.

Answer 1

Hi @Erica de Graaf-Borkent , it looks like the issue you are facing with your Dataflow connecting to CosmosDB may be due to an invalid or malformed subscription identifier.

This usually results from incorrect formatting or a misconfiguration in the linked service. Although you mentioned that the connection test was successful, the error message DFExecutorUserError suggests the problem may lie within the source rather than the linked service itself.

Please ensure that the linked service is properly configured, particularly the account name and endpoint. If you're using parameters that refer to another linked service, try temporarily removing those parameters to isolate the issue. It may also help to reimport the projection from the source and double-check that the data types in your sink match the schema of your CosmosDB collection. A mismatch in schema is a common cause of such errors.

Following some best practices can help avoid similar issues in the future: regularly clean up duplicate columns after transformations, use select statements to keep datasets lean, and always apply schema projection at the source, especially when changes occur. If the target CosmosDB collection already exists, make sure its schema is aligned with the sink configuration. Let us know if these steps help resolve the issue.

Erica de Graaf-Borkent 0 Reputation points

2025-04-25T13:01:20.95+00:00

I think my linked service is correctly configured, because when I use the Data flow debug mode my Data flow shows all steps correctly, with a preview of the data from the CosmosDB I expected to see.
I didn't use the system assigned identity in a POC I did before with the same CosmosDB, I used the account key to configure the linked service. And in that case my pipeline did run successfully.
Dileep Raj Narayan Thumula 0 Reputation points Microsoft External Staff

2025-04-28T08:13:20.63+00:00

@Erica de Graaf-Borkent Could you check whether the system assigned managed identity has proper permissions to the dataflow / ADF or not. "Grant the system-assigned managed identity proper permission.

Share via

AZF Dataflow error connection CosmosDB

1 answer

Your answer