Questions on Databricks Service Principal Login and ADF Pipeline Notebook Execution Identity
Hi Team, I have questions regarding Azure Data Factory (ADF) integration to databricks through SP Running Databricks Notebooks from ADF pipeline When I trigger a pipeline in ADF that runs a Databricks notebook, the notebook seems to execute under a…
Azure Data Factory
In our transformation processes (dataflow) in ADF, we are getting this error "Hit unexpected exception, please retry. If the problem persists, please contact Azure Support"
This has been happening for the last 7 days. Anyone else o just me?, The services are in East US 2. Thanks.
Azure Data Factory
I created a Private Link for ADF test in my MSDN subscription. Even after deleting all resources I am incurring cost
I created a private link to test a DB sync scenario using Azure Data Factory. After testing I deleted all the resources in my MSDN subscription. In spite of that I am incurring monthly charges up to $24. I tried to find that resource using Graph query,…
Azure Data Factory
ADF Linked service upgrade from Snowflake V1 to V2
Hello team, We are using Azure data factory for loading data to snowflake. We are trying to update the linked service from Snowflake V1 to Snowflake V2. I tried to use the connector upgrade using below process…
Azure Data Factory
Update Oracle Connection version 2.0 from 1.0
This is connected to another question I have. https://learn.microsoft.com/en-us/answers/questions/5490043/adf-oracle-connection-deprecated-version-1-0 Since the version 1 will be decommissioned soon. We need to update our Oracle connections to version 2.…
Azure Data Factory
My Azure Data factory is not opening.
My Azure Data factory is not opening. it is taking more time than excepted, still just buffering.
Azure Data Factory
Managed Airflow scheduler not working.
We have a managed Airflow instance in our subscription, and the UI shows that the last heartbeat was a day ago (08/09). I tried resetting the Airflow instance multiple times, either by changing a variable or importing dags from a storage, and nothing…
Azure Data Factory
SFTP connection problem from Azure Data Factory to a server that only supports rsa-sha2-256/512
Hello, I am trying to connect from Azure Data Factory (both with Managed IR and Self-Hosted IR) to the SFTP server of a supplier. But i receive the following error: "Failed to connect to Sftp server 'xxxxxx'. The connection was closed by the…
Azure Data Factory
ADF Snowflake V2 connector Issue
We are facing an issue while using ADF Snowflake V2 Connector as below ErrorCode=SnowflakeBuildConnectionError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Snowflake connection build failed with error: 'Error: Snowflake…
Azure Data Factory
consistency
transaction database guarantees consistency , all data changes will be logged and accounted for . is yes or NO
Azure Data Factory
Troubleshooting File Copy Failures to Azure Blob Storage
What could be causing certain data files to fail to copy to Azure Blob Storage while other files copy without issues? PS - Based on common issues that we have seen from customers and other sources, we are posting these questions to help the Azure…
Azure Data Factory
Help design an Azure native data pipeline (HTTP GET/Data Lake/compressed binary/SQL serving)
I need advice on an Azure only architecture to ingest data from a third-party tool and make it queryable for data scientists. Scenario Source exposes an HTTP endpoint accessible only via GET. Land the payloads in a data lake (Azure Blob Storage or…
Azure Data Factory
ADF v2 - Is there documentation listing all available properties for all pipeline Activities?
I've scoured the internet and cannot seem to find a documented list of all of the properties you can reference for any Azure Data Factory activity. I'm "sure" Microsoft published this list somewhere but I need help finding it. For example, I…
Azure Data Factory
what is sink in data pipeline?
Hi, Can someone please help to explain what sink is in data pipeline? is it used by default? if applicable, when to turn it on/off?
Azure Data Factory
Data factory pipeline execution fails on SqlParameter
Since a couple weeks our data factory pipelines sometimes fail due to below error message. We are running for approx 1,5 year in production and never experienced this error. So it is really strange why this happens. Also it does not happen each run,…
Azure Data Factory
How to move data into Data Lake Tables with Data Factory
We have an Azure Data Lake set up and this data lake will house data from multiple sources. I know how to use Data Factory to send data files/tables to the blob containers, but I want to store them in the tables instead. This data will be access and…
Azure Data Factory
copy activity fails in ADF
We are copying data from on-prem Oracle to an Azure SQL object. The pipeline is scheduled to trigger daily at 05:00 PM IST. The pipeline was running successfully until Wednesday, but from Thursday onwards it has been failing with the mentioned error.…
Azure Data Factory
AppInsights Kusto API [/v1/rest/query] Issue
We are noticing some intermittent issues in ADF when using partition based copy (using ForEach loop) for copying data from AppInsights to kusto tables. We see following error being thrown when processing some partitions (API /v1/rest/query fails): FYI -…
Azure Data Factory
ADF - Git configuration failure
What the permission is to initially Configure a repository with Azure DevOps? I got the error below:
Azure Data Factory
ADF Pipeline fails with error msg about TLS
I have a pipeline in ADF that runs an SSIS package. It occasionally fails with a message about the Client TLS version: "Login failed due to client TLS version being less than minimal TLS version allowed by the server" I can successfully run…