10,599 questions with Azure Data Factory tags

Sort by: Updated
1 answer

Access issue with DB post deployment

Hi, I just learned how to do CI/CD for ADF and am trying out a test. I have the sql db's in a different Resource group (rg_sql_dev) and the ADF's resource group is RG_ADF_DEV. Likewise, for the production i have it rg_sql_prod and RG_ADF_PROD I created…

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-20T19:55:54.6433333+00:00
sam nick 286 Reputation points
edited an answer 2024-09-21T08:13:33.1066667+00:00
Vinodh247 19,226 Reputation points
0 answers

Power Query merge query on two colums

Hello: I have two datasets (UserQueries) in Power Query, I now want to merge them on two columns that are common. Should be easy, right. I use Merge Queries on the tool bar, get a dialogue box as follows: The col's are correct and the order is correct.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-20T21:53:31.9466667+00:00
KBD 40 Reputation points
commented 2024-09-21T06:53:38.68+00:00
Vinodh247 19,226 Reputation points
1 answer

Salesforce Connector deprecation details

As part of Sales force legacy connector, can we get confirmation for below ask please: "You can continue to use existing Linked services and datasets of the legacy connector and make changes to them. You cannot create a new Linked service for the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-19T14:03:55.46+00:00
Narendra Kandula 0 Reputation points
commented 2024-09-20T17:26:43.5233333+00:00
Smaran Thoomu 15,040 Reputation points Microsoft Vendor
1 answer

Failed to get MI access token.

I have 25 pipelines running but all are failing from yesterday due to this error: ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Failed to get MI access token.…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,166 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
7,685 questions
asked 2024-02-21T12:59:58.12+00:00
SHARAD SONWANE 0 Reputation points
commented 2024-09-20T17:25:20.9266667+00:00
PK 20 Reputation points
0 answers

Error on sink data to MySQL - communications link failure

I've got a Pipeline with some Data Flow activities that runs in parallel. I've got the error on last night's execution and I'd like some help to understand why and how to prevent it from happenning. Job failed due to reason: at Sink 'EnviaFerias':…

Azure Database for MySQL
Azure Database for MySQL
An Azure managed MySQL database service for app development and deployment.
819 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-19T11:24:37.3633333+00:00
commented 2024-09-20T17:24:57.4366667+00:00
Smaran Thoomu 15,040 Reputation points Microsoft Vendor
2 answers

Pipeline run stucked on queue

I have some pipelines that ingest data from Oracle and write directly to Delta in a lakehouse on Fabric. I have other pipelines that ingest data from Oracle and write to a .parquet file in a lakehouse on Fabric. The pipelines that write directly to Delta…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-16T14:11:07.89+00:00
Gabriel Santos 0 Reputation points
commented 2024-09-20T17:23:12.3166667+00:00
Smaran Thoomu 15,040 Reputation points Microsoft Vendor
6 answers

Azure Data Factory Until loop slow to end/terminate

We have an until loop in a ADFv2 pipeline. The time it takes to stop/terminate once the expression condition is met seems to corrolate between the length of time the until loop takes to completes its activities. This particular until loop…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2022-10-10T07:38:06.123+00:00
Geezer 26 Reputation points
commented 2024-09-20T15:22:40.4366667+00:00
Bäcker 21 Reputation points
1 answer

ADF logging mechanism

I have a few options to get logging from Azure data factory for performing key metric analysis such as Having SQL table store procedures and capture all key metrics from Azure monitor such pipeline runtimes and durations Programmatically grap these…

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,216 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-20T14:35:25.8066667+00:00
azure_learner 240 Reputation points
answered 2024-09-20T15:06:01.3466667+00:00
Nandan Hegde 32,026 Reputation points MVP
1 answer

Azure Synapse pipeline copy data activity from on-prem to Azure sql db issue

{ "errorCode": "2200", "message": "ErrorCode=SqlFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot connect to SQL Database. Please contact SQL server team for further…

Azure SQL Database
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,877 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-17T17:23:29.7766667+00:00
Madhu Challapalli 20 Reputation points
commented 2024-09-20T14:59:38.7233333+00:00
Madhu Challapalli 20 Reputation points
1 answer

Azure data lake folder structure

Hi, the link provided in the below thread is not working, any way to find the information or a new URL/link: FAQs About Organizing a Data…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,466 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-19T08:43:49.7+00:00
azure_learner 240 Reputation points
answered 2024-09-20T14:26:26.4+00:00
azure_learner 240 Reputation points
0 answers

Json response could not parse successfully in Copy activity (source)

For the same HTTP request, Web activity gives us the correct response all correctly parse, but Copy activity couldnt. This has worked fine before. Screenshots are cut from Preview data. When I actually run the copy activity there is no data coming from…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-19T14:55:37.5466667+00:00
Quyen Dang 21 Reputation points
commented 2024-09-20T13:36:07.9566667+00:00
phemanth 10,255 Reputation points Microsoft Vendor
0 answers

The specified network name is no longer available - error code - 22852

Hello Team, Today I faced an error, when i am trying to do test connection. Please resolved as soon as possible. I need to resolve please help me to resolve. 

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-20T12:10:29.63+00:00
Balwant Singh Chouhan 0 Reputation points
commented 2024-09-20T13:27:48.5933333+00:00
Smaran Thoomu 15,040 Reputation points Microsoft Vendor
1 answer

I am trying to get ADF to connect to Microsoft Purview.

Azure is having a new UI. I am not able to activate the ADF flow to push metadata to Microsoft purview. The documentation explains the following: I don't even see Management option in Microsoft Purview. Follow the steps below to connect an existing…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
Microsoft Purview
Microsoft Purview
A Microsoft data governance service that helps manage and govern on-premises, multicloud, and software-as-a-service data. Previously known as Azure Purview.
1,144 questions
asked 2024-09-16T19:25:58.5633333+00:00
Ram Veera 0 Reputation points
commented 2024-09-20T13:26:47.4366667+00:00
Ram Veera 0 Reputation points
0 answers

Connecting Azure Data factory to Oracle ADW

Hi, While trying to connect to Oracle ADW using Wallet file, facing the below error, how to resolve ? Tried: https://learn.microsoft.com/en-ca/azure/data-factory/connector-oracle-service-cloud?tabs=data-factory#linked-service-properties ERROR [HY000]…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-19T23:35:41.7566667+00:00
Priya Chary 0 Reputation points
commented 2024-09-20T12:06:52.2266667+00:00
AnnuKumari-MSFT 32,821 Reputation points Microsoft Employee
0 answers

Failing to copy large dataset from Salesforce using new salesforce V2 connector in Azure data factory

I am trying to migrate the salesforce legacy linked service to new salesforce V2 linked service in Azure data factory. The test connection is successful and I am able to copy the smaller datasets using new salesforce V2 linked service. But when i am…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-17T23:02:44.6066667+00:00
LY 0 Reputation points
commented 2024-09-20T11:48:15.0233333+00:00
phemanth 10,255 Reputation points Microsoft Vendor
0 answers

gRPC connection errors in copy activity

I have a copy data activity in ADF that copies from Google BigQuery to Blob storage. Occasionaly I get one of these errors: "grpc_message":"End of TCP stream" "grpc_message":"keepalive watchdog timeout" We are…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-20T07:24:28.6333333+00:00
Hamza Outa 61 Reputation points
commented 2024-09-20T11:23:22.6933333+00:00
phemanth 10,255 Reputation points Microsoft Vendor
2 answers

Why am I unable to create a cluster in Azure databricks?

Currently, I am using a student account. I tried selecting different configurations. It would be great if someone recommend a configuration for student account!

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,166 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-14T21:34:53.8233333+00:00
mksurabh 0 Reputation points
commented 2024-09-20T10:51:58.56+00:00
PRADEEPCHEEKATLA-MSFT 88,951 Reputation points Microsoft Employee
1 answer

adf/synapse - how to copy all files based on given lastmodified date from source container and save in sink container in respective folder based on extension

Req: Copy files from one container and store in another container- I am specifying the last modified date , i want to copy files only before that last last modified date. The files are of any type. csv,xml,json,txt etc. Also, in source container my files…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,877 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-09-17T20:33:13.42+00:00
Wasim 0 Reputation points
commented 2024-09-20T08:25:41.4266667+00:00
Vinodh247 19,226 Reputation points
1 answer

facing HTTP request was forbidden for 365 Dynamics linked service

The HTTP request was forbidden with client authentication scheme 'Anonymous'. The remote server returned an error: (403) Forbidden.

Microsoft 365
Microsoft 365
Formerly Office 365, is a line of subscription services offered by Microsoft which adds to and includes the Microsoft Office product line.
4,775 questions
Microsoft Graph
Microsoft Graph
A Microsoft programmability model that exposes REST APIs and client libraries to access data on Microsoft 365 services.
11,893 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
asked 2024-08-27T11:49:08.9333333+00:00
Vaibhav Patil 235 Reputation points
edited the question 2024-09-20T07:57:20.3+00:00
Rakesh Gurram 7,085 Reputation points Microsoft Vendor
1 answer

Dynamic Mapping of Source to JSON Sink in Azure Data Factory

We are able to preview the source data from the Salesforce linked service in ADF, and I can also view the sink data using the REST linked service for Business Central in ADF. However, I'm facing an issue where importing the schema is not supported…

Microsoft Graph
Microsoft Graph
A Microsoft programmability model that exposes REST APIs and client libraries to access data on Microsoft 365 services.
11,893 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,877 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,599 questions
Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
21,476 questions
asked 2024-09-06T12:01:43.23+00:00
Apoorva Belsare 0 Reputation points
edited the question 2024-09-20T07:48:34.3533333+00:00
Rakesh Gurram 7,085 Reputation points Microsoft Vendor