11,633 questions with Azure Data Factory tags

Sort by: Updated
1 answer

How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage

Hi Team, I would like to know following questions. Please do needful. 1.How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage 2.How to connect Hadoop hive to databricks Regards, Prasad, 5017654887.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-07-03T02:41:39.2233333+00:00
Prasad Sandu 0 Reputation points
answered 2025-07-03T08:32:44.51+00:00
Amira Bedhiafi 33,316 Reputation points Volunteer Moderator
3 answers

How to iterate through Nested folders and read file after the file in adf

Hi, How can I iterate through folder and inside subfolder by folder and process one file after another without using regex in azure data factory

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-07-02T13:02:06.2633333+00:00
Radha Jahnavi Pusarapu 0 Reputation points
commented 2025-07-03T04:52:42.1933333+00:00
Radha Jahnavi Pusarapu 0 Reputation points
3 answers

How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute

Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,521 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2024-05-01T12:12:06.9033333+00:00
Krzysztof Przysowa 20 Reputation points
answered 2025-07-03T04:28:23.1366667+00:00
Yunpeng Tang 0 Reputation points Microsoft Employee
1 answer

How does Azure Data Factory handle data movement ?

How does Azure Data Factory handle data movement and transformation across hybrid environments, and what are the best practices for optimizing pipeline performance and cost efficiency

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-07-02T22:05:28.1966667+00:00
Himanshu Sinha 11 Reputation points
answered 2025-07-02T23:17:32.56+00:00
Smaran Thoomu 24,260 Reputation points Microsoft External Staff Moderator
1 answer

Seeking solution for intermittent "SqlFailedToConnect" errors when writing to Azure SQL Managed Instance from Azure Data Factory

I have multiple ADF pipelines that copy data from a variety of sources into our Azure SQL Managed Instance. All of these jobs, regardless of source, fail intermittently with a Sink error such…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-07-02T18:12:10.1166667+00:00
Rob Lamb 0 Reputation points
answered 2025-07-02T19:05:24.24+00:00
Amira Bedhiafi 33,316 Reputation points Volunteer Moderator
1 answer One of the answers was accepted by the question author.

dataset is using 'AzureFileStorage' linked service type,which is not supported in dataflow. How to resolve this issue?

dataset is using 'AzureFileStorage' linked service type,which is not supported in dataflow. How to resolve this issue?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-07-02T06:02:32.1033333+00:00
Tom Abraham 0 Reputation points
commented 2025-07-02T12:43:07.2733333+00:00
Tom Abraham 0 Reputation points
1 answer

data extraction issues since upgrading to Odata 4.0 in ADF

Our data extraction process, using the Odata connector in ADF has shown problems since our client upgraded their service to Odata 4.0. The issues consist of not succesfully extracting an endpoint. Our client shows that their network gets overloaded even…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-07-01T09:34:47.71+00:00
Thijs Bakker 0 Reputation points
commented 2025-07-02T10:41:26.95+00:00
Venkat Reddy Navari 3,060 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

Can I use a KeyVault Connection String with the Azure Data Factory Mysql Linked Service Driver v2.0?

Hi all, I currently use a V1.0 MySQL linked services as part of my Azure Data Factory setup. I use a secret connection string stored in Azure Key Vault. I am being asked to upgrade the linked service driver to 2.0, but it seems this does not support the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-07-02T08:34:26.1233333+00:00
Peter Rush 41 Reputation points
accepted 2025-07-02T10:39:36.35+00:00
Peter Rush 41 Reputation points
0 answers

ServiceNow Incident table extraction with V2 Connectors

Using the new ServiceNowV2 connectors in Azure Data Factory, we are trying to extract the incident table. When setting up a copy activity, we receive the error message below. I have tried to limit the number of records by providing a filter to small…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-07-01T06:53:26.65+00:00
Joshoua Simon 0 Reputation points
commented 2025-07-02T10:24:21.19+00:00
Venkat Reddy Navari 3,060 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

Conditional Split in ADF Data Flow issue - no rows in sink

I have a Conditional Split transformation that works fine except one sink results empty although there are source rows matching the split condition. In my scenario I have - say - 4 conditions, two of which are supposed to end up in the same table sink.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-06-26T14:28:01.3266667+00:00
Ulrike Pick 20 Reputation points
accepted 2025-07-02T09:36:51.0866667+00:00
Ulrike Pick 20 Reputation points
1 answer

Azure Data Factory: No 'Name' Key Appearing Under Additional Headers For REST APIs Data Copy Action

Hello, I'm trying to bring in data to SQL Server using a REST API. I've successfully completed the first stage, where I needed to call the Login endpoint to fetch the apiToken in order to request all other endpoints using the bearer authentication. I'm…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-06-24T17:19:53.7366667+00:00
sk900 0 Reputation points
commented 2025-07-02T08:35:53.86+00:00
Smaran Thoomu 24,260 Reputation points Microsoft External Staff Moderator
1 answer

Failure happened on 'Source' side. Copy activity Error in Fabric

Failure happened on 'Source' side. ErrorCode=UserErrorWriteFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-06-27T00:32:51.13+00:00
Sri Ram Katta 0 Reputation points
commented 2025-07-01T16:29:24.6466667+00:00
Venkat Reddy Navari 3,060 Reputation points Microsoft External Staff Moderator
1 answer

I have added parameter in Dataset but it is not reflecting in copy activity

I have added parameter in Dataset but it is not reflecting in copy activity while coping all sql tables form azure sql to blob storage , created 2 parameter (schema,table) for pass lookup output values

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-06-30T07:39:03.91+00:00
Rupesh Boiwar 0 Reputation points
commented 2025-07-01T16:26:38.2366667+00:00
Venkat Reddy Navari 3,060 Reputation points Microsoft External Staff Moderator
2 answers

How to make the Chekpoint Key / SAP to stage subscriber process a static value?

Context: I am currently building a pipeline that extracts data from a SAP CDC source, where I want to utilize the delta mechanism this connector enables. My setup is: SAP CDC connector to source system (SAP CRM)- static subscriber name and ODP…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2023-12-09T11:16:12.3066667+00:00
Georgi Ivanov 0 Reputation points
answered 2025-07-01T13:51:58.1933333+00:00
Bröthaler Thomas 0 Reputation points
1 answer One of the answers was accepted by the question author.

Compatibility and Support for Amazon Corretto in Azure Data Factory

Hello Azure Support Team, We are exploring the possibility of replacing Oracle Java with Amazon Corretto (OpenJDK distribution) in our Azure Data Factory (ADF) environment. Before proceeding, we would like to clarify the following: Compatibility: Is…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-06-23T13:27:38.9433333+00:00
Naveen Prasanth 20 Reputation points
accepted 2025-07-01T10:38:33.73+00:00
Naveen Prasanth 20 Reputation points
3 answers

AZF Dataflow error connection CosmosDB

I have created a linked service and the testconnection is succeeded But when I use this linked service in a dataflow and test the connection I get the following error: Error codeDFExecutorUserError DetailsStatus code 400,…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-04-22T11:34:58.58+00:00
Erica de Graaf-Borkent 0 Reputation points
answered 2025-07-01T07:37:39.9133333+00:00
Babu Kandyala 0 Reputation points
2 answers One of the answers was accepted by the question author.

Issue with "Too Many Requests" Error in App Provisioning

I ran the code in the test file within Synapse and observed a "Too Many Requests" error while processing records for App provisioning. We also confirmed this issue via Postman, where the same error was encountered.   Can anyone help me to…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
8,953 questions
asked 2024-12-04T14:42:12.2733333+00:00
Vaibhav Patil 255 Reputation points
edited the question 2025-07-01T05:22:13.5133333+00:00
Chandra Boorla 14,585 Reputation points Microsoft External Staff Moderator
1 answer

What are the recommended steps for setting up a Service Principal in Azure for enterprise use, and how should the IT team manage its lifecycle securely?

We’re working on setting up Service Principals (App Registrations) in Azure to support automated data workflows (e.g., Azure Data Factory accessing Azure SQL, ADLS, etc.). The goal is to enable secure, non-interactive authentication while following…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-06-27T09:33:08.0033333+00:00
YERNAIDU SIRAPARAPU 45 Reputation points
commented 2025-06-30T06:35:17.3033333+00:00
Venkat Reddy Navari 3,060 Reputation points Microsoft External Staff Moderator
1 answer

'json' parameter is not valid after updating secret key in ADF web activity

Hello, We have a pipeline with two web activities: The first retrieves a secret key from the key vault. The second uses that secret key to obtain a bearer token. Recently, we updated the expired secret key retrieved in the first web activity. The…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-03-20T15:33:29.6633333+00:00
Anneka Osmun 0 Reputation points
commented 2025-06-30T01:59:11.85+00:00
J N S S Kasyap 3,700 Reputation points Microsoft External Staff Moderator
1 answer

recommended SHIR configuration to ingest 80TB from DB2 Snapshot to ADLS Gen2 via ADF within 3 days?

We are performing a large-scale one-time data migration project. Our source is an IBM DB2 snapshot (on-prem), and the target is Azure Data Lake Storage Gen2. The entire data volume is ~80TB, distributed across 800 tables. We are using Azure Data Factory…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2025-06-28T15:53:10.39+00:00
Janice Chi 140 Reputation points
commented 2025-06-30T01:41:45.2233333+00:00
Venkat Reddy Navari 3,060 Reputation points Microsoft External Staff Moderator