Share via

ADF CI/CD exceeding maximum number of template parameters

Question

Wednesday, June 26, 2019 11:39 PM | 2 votes

 I followed the instructions for scaling resource group deployments for CI/CD for an ADF factory.

I have a bunch of 30+ files that are triggered by blob events.  They cause a copy activity.  So each source dataset, trigger and destination dataset will have about 3 parameters.  This limits us to around 30 file type otherwise we go over the 256 parameter limit when deploying the template through devops. (note - parameters - not resources).

when deploying an additional file type, it complained about the number of parameters (not resources). 

2019-06-23T04:39:43.0617813Z The detected encoding for file 'C:\agent\_work\r3\a\_ADF03_Publish\xxxFACTORY03-DEV\linkedTemplates\ArmTemplate_master.json' is 'utf-8'
2019-06-23T04:39:43.0853688Z The detected encoding for file 'C:\agent\_work\r3\a\_ADF03_Publish\xxxDFACTORY03-DEV\linkedTemplates\ArmTemplateParameters_master.json' is 'utf-8'
2019-06-23T04:39:43.1037033Z Starting Deployment.
2019-06-23T04:39:43.1037730Z Deployment name is ArmTemplate_master-20190623-043943-8b6b
2019-06-23T04:39:43.1745042Z There were errors in your deployment. Error code: InvalidTemplate.
2019-06-23T04:39:43.1745370Z ##[error]Deployment template validation failed: 'The number of template input parameters limit exceeded. Limit: '256' and actual: '259'. Please see https://aka.ms/arm-template/#parameters for usage details.'.
2019-06-23T04:39:43.1745751Z ##[error]Task failed while creating or updating the template deployment.

Is there a work around when the number of parameters (256) are exceeded ?

Splitting into two factories is a pain as we are running on linked integration runtimes (adding to the complexity).

Thanks, MArk.

All replies (4)

Thursday, June 27, 2019 1:38 AM

/en-us/azure/data-factory/continuous-integration-deployment#linked-resource-manager-templates

Please take a look at above doc.

If you've set up continuous integration and deployment (CI/CD) for your Data Factories, you may observe that, as your factory grows bigger, you run into the Resource Manager template limits, like the maximum number of resources or the maximum payload in a Resource Manager template. For scenarios like these, along with generating the full Resource Manager template for a factory, Data Factory also now generates Linked Resource Manager templates. As a result, you have the entire factory payload broken down into several files, so that you don’t run into the mentioned limits.

Thursday, July 4, 2019 9:16 AM

Not the correct answer, he is talking about the input parameter exceeded.

Thursday, July 11, 2019 12:07 AM | 1 vote

Hi Mark,

Since it is limited to 256 parameters in a template, you can reduce the number of parameters by using objects that contain multiple properties. For additional info please refer to below docs, which might help you.

Additional info:

  • Custom parameters are useful when your data factory has grown so huge that you exceed the 256 parameter limit.

Hope this helps.

Thank you.

[If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster. ]

Thursday, December 19, 2019 7:37 PM

The workaround to use objects that contain multiple properties works only once after manually modifying the templates generated by ADF, but when publishing from ADF again, the templates are overwritten and it will hit the parameters limit again.

Is there any workaround to make ADF generate templates with fewer parameters or using the object parameters so it will work in a CI/CD scenario?

Thanks!

  • Last updated on