How to parametrize in the arm-template-parameters-definition.json the referenceName for Linked Service Reference in the Datasets part.

Question

How to parametrize in the arm-template-parameters-definition.json the referenceName for Linked Service Reference in the Datasets part.

Dina 0

I would like to parametrize in the arm-template-parameters-definition.json the "referenceName" for linked service in the datasets part of the file. You can have a look at the below image the part I want to change in order the production environment to take the correct changes:

User's image

As you can see i tried to add the referenceName with the equal symbol "=" but it did not let me change the value for prod environment. I also tried the minus symbol "-" but it didn't work either. How it should be parametrized in order to have the ability to change the value for the prod environment in the pipeline?

Thank you

Ganesh Gurram 7,235 Reputation points Moderator

2025-03-11T04:47:27.6666667+00:00
Hi @Dina

To parameterize the referenceName for a linked service in the datasets section of arm-template-parameters-definition.json, please follow these steps:

Modify arm-template-parameters-definition.json

Define the parameter using the = action to allow environment-specific values:

{ "Microsoft.DataFactory/factories/datasets": { "*": { "properties": { "linkedServiceName": { "referenceName": "=linkedServiceReferenceName" } } } } }

The = action keeps the current value as default but allows overriding.

"linkedServiceReferenceName" is the parameter name that will be set dynamically.

Define the Parameter in arm-template-parameters.json

Ensure the parameter is included in the deployment parameters file:

{ "$schema": "https://schema.management.azure.com/providers/Microsoft.Resources/deploymentParameters?api-version=2019-10-01", "contentVersion": "1.0.0.0", "parameters": { "linkedServiceReferenceName": { "value": "YourLinkedServiceNameForProd" } } }

Replace "YourLinkedServiceNameForProd" with the correct value for the production environment.

Update Your Dataset JSON File

Ensure that the dataset definition references the parameter correctly:

{ "name": "YourDatasetName", "properties": { "linkedServiceName": { "referenceName": "[parameters('linkedServiceReferenceName')]" } } }

Verify that there are no hardcoded values for referenceName in your dataset JSON.

Confirm that your deployment pipeline references the correct arm-template-parameters.json

For more details, refer to:

Use custom parameters with the Resource Manager template

Parameterize linked services in Azure Data Factory and Azure Synapse Analytics

Hope this helps. Do let us know if you have any further queries.
Dina 0 Reputation points

2025-03-11T10:12:58.2+00:00

Hello Ganesh and thank for your response but unfortunately didn't work.

I followed your feedback by trying this in my branch in the dev environment:

When I exported the arm template in order to check it in my branch I couldn't see any dataset in the ARMTamplateParametersForFactory.json in order to be able to change it later in the pipeline. I also tried to put the "=linkedServiceReference" with capital "L" like this "=LinkedServiceReference" but also didn't work.

Is there any mistake I did?

What should I try now?

Thank you
Ganesh Gurram 7,235 Reputation points Moderator

2025-03-12T08:30:22.4266667+00:00
@Dina - I understand that the dataset parameters did not appear in ARMTemplateParametersForFactory.json after exporting the ARM template.

Ensure that the dataset parameterization follows the correct structure:

{ "Microsoft.DataFactory/factories/datasets": { "*": { "properties": { "linkedServiceName": { "referenceName": "=datasets_<DatasetName>_properties_linkedServiceName_referenceName" } } } } }

Replace <DatasetName> with the actual dataset name.

Ensure there are no typos (e.g., case-sensitive issues in LinkedServiceReference).

If the dataset parameter is missing from the exported file - Ensure that the dataset is correctly referenced in your Data Factory pipeline. If referenceName is hardcoded in the dataset JSON, it may prevent the parameter from being generated.

I hope this information helps!
Dina 0 Reputation points

2025-03-12T14:38:13.62+00:00

It did not work again. I did what you said. Also I have many datasets that I want to change the value of the linked service reference in the prod environment, not just one.

Unfortunately there is no way to put dynamic content in the linkedServiceReference inside the dataset.

It give you the possibility of only a default selection of linked services. The thing is that when I parametrized the linked services a lot properties like "client_id", "baseUri", "databaseName" etc it worked, although I don't have any dynamic parametrization. Also when I parametrized some pipelines that have the type of AzureMLExecutePipeline and I wanted to parametrize the experimentName and mlPipelineEndpointId, the parametrization was successful , but now in the datasets it doesn't work the linked service reference part.

Also in the below image I am giving you just one dataset of the ones I have to parametrize in order for you to see the content and context. As you can see it is not dynamic the part of linked service it is just hardcoded the linked service and as I told you there is no way to parametrize it(it doesn't show a dynamic content in order to put parameters).

Is there any chance to have another solution? Can you give me all the steps in series that I should for this thing to succeed in case I forgot something?
Anonymous

2025-03-14T06:52:05.57+00:00
Hello @Dina,

It looks like the referenceName for linkedServiceName in datasets is not supporting parameterization directly within the arm-template-parameters-definition.json file, unlike other properties you successfully parameterized.

Since datasets are resource objects, the recommended approach is to parametrize the dataset itself, rather than just modifying arm-template-parameters-definition.json.

Instead of dynamically assigning referenceName via arm-template-parameters-definition.json, you need to modify the dataset JSON itself to accept parameters.

1. Modify the Dataset JSON to Use Parameters

In each dataset JSON definition (dataset.json in the ARM template), replace the hardcoded referenceName with a parameter reference:

{ "name": "YourDataset", "properties": { "linkedServiceName": { "referenceName": "[parameters('LinkedServiceName')]", "type": "LinkedServiceReference" }, "typeProperties": { "location": { "type": "AzureFileStorageLocation", "folderPath": "[parameters('FolderPath')]" } }, "compressionCodec": "snappy" } }

2. Define the Parameters in arm-template-parameters.json

Now, you need to define LinkedServiceName for different environments:

{ "parameters": { "LinkedServiceName": { "value": "YourLinkedServiceForDev" }, "FolderPath": { "value": "some/path" } } }

For Prod, create a separate parameter file (arm-template-parameters-prod.json):

{ "parameters": { "LinkedServiceName": { "value": "YourLinkedServiceForProd" }, "FolderPath": { "value": "some/prod/path" } } }

Deploy the Template with Environment-Specific Parameters.

Hope this helps. Do let us know if you have any further queries.
Anonymous

2025-03-17T03:36:11.0866667+00:00

@Dina, we haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Anonymous

2025-03-18T12:11:01.6166667+00:00

@Dina, we haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Ganesh Gurram 7,235 Reputation points Moderator

2025-03-11T04:47:27.6666667+00:00

Hi @Dina

To parameterize the referenceName for a linked service in the datasets section of arm-template-parameters-definition.json, please follow these steps:

Modify arm-template-parameters-definition.json

Define the parameter using the = action to allow environment-specific values:

{ "Microsoft.DataFactory/factories/datasets": { "*": { "properties": { "linkedServiceName": { "referenceName": "=linkedServiceReferenceName" } } } } }

The = action keeps the current value as default but allows overriding.

"linkedServiceReferenceName" is the parameter name that will be set dynamically.

Define the Parameter in arm-template-parameters.json

Ensure the parameter is included in the deployment parameters file:

{ "$schema": "https://schema.management.azure.com/providers/Microsoft.Resources/deploymentParameters?api-version=2019-10-01", "contentVersion": "1.0.0.0", "parameters": { "linkedServiceReferenceName": { "value": "YourLinkedServiceNameForProd" } } }

Replace "YourLinkedServiceNameForProd" with the correct value for the production environment.

Update Your Dataset JSON File

Ensure that the dataset definition references the parameter correctly:

{ "name": "YourDatasetName", "properties": { "linkedServiceName": { "referenceName": "[parameters('linkedServiceReferenceName')]" } } }

Verify that there are no hardcoded values for referenceName in your dataset JSON.

Confirm that your deployment pipeline references the correct arm-template-parameters.json

For more details, refer to:

Use custom parameters with the Resource Manager template

Parameterize linked services in Azure Data Factory and Azure Synapse Analytics

Hope this helps. Do let us know if you have any further queries.
Dina 0 Reputation points

2025-03-11T10:12:58.2+00:00

Hello Ganesh and thank for your response but unfortunately didn't work.

I followed your feedback by trying this in my branch in the dev environment:

When I exported the arm template in order to check it in my branch I couldn't see any dataset in the ARMTamplateParametersForFactory.json in order to be able to change it later in the pipeline. I also tried to put the "=linkedServiceReference" with capital "L" like this "=LinkedServiceReference" but also didn't work.

Is there any mistake I did?

What should I try now?

Thank you
Ganesh Gurram 7,235 Reputation points Moderator

2025-03-12T08:30:22.4266667+00:00

@Dina - I understand that the dataset parameters did not appear in ARMTemplateParametersForFactory.json after exporting the ARM template.

Ensure that the dataset parameterization follows the correct structure:

{ "Microsoft.DataFactory/factories/datasets": { "*": { "properties": { "linkedServiceName": { "referenceName": "=datasets_<DatasetName>_properties_linkedServiceName_referenceName" } } } } }

Replace <DatasetName> with the actual dataset name.

Ensure there are no typos (e.g., case-sensitive issues in LinkedServiceReference).

If the dataset parameter is missing from the exported file - Ensure that the dataset is correctly referenced in your Data Factory pipeline. If referenceName is hardcoded in the dataset JSON, it may prevent the parameter from being generated.

I hope this information helps!
Dina 0 Reputation points

2025-03-12T14:38:13.62+00:00

It did not work again. I did what you said. Also I have many datasets that I want to change the value of the linked service reference in the prod environment, not just one.

Unfortunately there is no way to put dynamic content in the linkedServiceReference inside the dataset.

It give you the possibility of only a default selection of linked services. The thing is that when I parametrized the linked services a lot properties like "client_id", "baseUri", "databaseName" etc it worked, although I don't have any dynamic parametrization. Also when I parametrized some pipelines that have the type of AzureMLExecutePipeline and I wanted to parametrize the experimentName and mlPipelineEndpointId, the parametrization was successful , but now in the datasets it doesn't work the linked service reference part.

Also in the below image I am giving you just one dataset of the ones I have to parametrize in order for you to see the content and context. As you can see it is not dynamic the part of linked service it is just hardcoded the linked service and as I told you there is no way to parametrize it(it doesn't show a dynamic content in order to put parameters).

Is there any chance to have another solution? Can you give me all the steps in series that I should for this thing to succeed in case I forgot something?
Anonymous

2025-03-14T06:52:05.57+00:00

Hello @Dina,

It looks like the referenceName for linkedServiceName in datasets is not supporting parameterization directly within the arm-template-parameters-definition.json file, unlike other properties you successfully parameterized.

Since datasets are resource objects, the recommended approach is to parametrize the dataset itself, rather than just modifying arm-template-parameters-definition.json.

Instead of dynamically assigning referenceName via arm-template-parameters-definition.json, you need to modify the dataset JSON itself to accept parameters.

1. Modify the Dataset JSON to Use Parameters

In each dataset JSON definition (dataset.json in the ARM template), replace the hardcoded referenceName with a parameter reference:

{ "name": "YourDataset", "properties": { "linkedServiceName": { "referenceName": "[parameters('LinkedServiceName')]", "type": "LinkedServiceReference" }, "typeProperties": { "location": { "type": "AzureFileStorageLocation", "folderPath": "[parameters('FolderPath')]" } }, "compressionCodec": "snappy" } }

2. Define the Parameters in arm-template-parameters.json

Now, you need to define LinkedServiceName for different environments:

{ "parameters": { "LinkedServiceName": { "value": "YourLinkedServiceForDev" }, "FolderPath": { "value": "some/path" } } }

For Prod, create a separate parameter file (arm-template-parameters-prod.json):

{ "parameters": { "LinkedServiceName": { "value": "YourLinkedServiceForProd" }, "FolderPath": { "value": "some/prod/path" } } }

Deploy the Template with Environment-Specific Parameters.

Hope this helps. Do let us know if you have any further queries.
Anonymous

2025-03-17T03:36:11.0866667+00:00

@Dina, we haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Anonymous

2025-03-18T12:11:01.6166667+00:00

@Dina, we haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Hello again,

Thank you for your previous reply.

We didn't approach your recommended approach.

Unfortunately now we have a serious problem with parameters limit 256. We exceeded this value to 297.

The problem is with the datasets. We have 37 parquet datasets but only one of them uses a different a different linked on production env, that's why we have to parametrize the parquet datasets. At the moment this is the current configuration of parquet datasets:

    "Microsoft.DataFactory/factories/datasets": {
        "*": {
            "properties": {}
        },
        "Parquet": {
            "properties": {
                "linkedServiceName": {
                    "referenceName": "=",
                    "type": "LinkedServiceReference"
                }
            }
        },
        "Json": {
            "properties": {
                "linkedServiceName": {
                    "referenceName": "=",
                    "type": "LinkedServiceReference"
                }
            }
        }
    },

As you can see the above approach parametrizes all the parquet type datasets. Is there a way to parametrize only the 1 datasets that changes its value from dev to prod? Can you please provide all the required steps? Is there a way to group all the other datasets that use a specific linked service?

How to parametrize in the arm-template-parameters-definition.json the referenceName for Linked Service Reference in the Datasets part.

1. Modify the Dataset JSON to Use Parameters

2. Define the Parameters in arm-template-parameters.json

1 answer

Your answer

2. Define the Parameters in `arm-template-parameters.json`