An Azure service for ingesting, preparing, and transforming data at scale.
ADF Custom Activity for Batch accounts intermittently throws error 'XML nodes is not in the correct format'
I have multiple pipelines that use ADF Custom Activity to Batch Account. They normally run successfully. However they were failing consistently for 18 hours yesterday with the following error " The value for one of the XML nodes is not in the…
Azure Data Factory
Storing REST API Response as Raw JSON in SQL Table Instead of Mapped Columns in Azure Data Factory
I am using Copy Activity in Azure Data Factory in which I am using REST API for getting the Response from the API I am getting an array contain multiple JSON. I wanted to Store the Response as a Raw Format in the SQL Table but In the Mapping I am getting…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Handling Nested JSON Arrays from Business Central APIs using Rest in Azure Data Factory – Best Approach?
I’m building a dynamic ingestion pipeline in Azure Data Factory to load data from Microsoft Business Central APIs into Azure SQL staging tables, and I’ve encountered several challenges related to the JSON structure returned by the API. A typical Business…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Copy Merchant Data, Error: Failure happened on 'Sink' side. ErrorCode=UserErrorDocumentDBWriteError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Documents failed to import. Error message:Missing partition key header for point
Operation on target ForEach failed: Activity failed because an inner activity failed; Inner activity name: Copy Merchant Data, Error: Failure happened on 'Sink' side.…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
ADF to Snowflake sync setup UPSERT
I Azure Data Factory pipeline setup to sync data into my Snowflake account. Currently I use Snowflake V2 connector, but in my Copy Data action I can't select it to be UPSERT as INSERT duplicates entries which is not correct in my case. How can I setup…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Dataflow UTF8 withoutBOM is wrong
According to the article https://learn.microsoft.com/en-us/azure/data-factory/format-delimited-text data flow doesn't support UTF-8 encoding with Byte Order Mark (BOM) but support UTF8 without bom. I made a test but the result diversed. I made dataset…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
How to fix: Request Id: fccd854a-cfc2-43a1-89dd-0403c2493700 Correlation Id: 4fb66761-4ec3-477e-8c3b-d9c2a5c8562e Timestamp: 2024-01-16T19:29:03Z Message: AADSTS900561: The endpoint only accepts POST requests. Received a GET request.
Request Id: fccd854a-cfc2-43a1-89dd-0403c2493700 Correlation Id: 4fb66761-4ec3-477e-8c3b-d9c2a5c8562e Timestamp: 2024-01-16T19:29:03Z Message: AADSTS900561: The endpoint only accepts POST requests. Received a GET request. I can´t login in the web anymore…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
SharePoint Online ACS Retirement - What's the correct way to migrate Azure Data Factory (ADF) SPO List Linked Services
As per the title, I couldn't find any way I can use ADF Linked Services to ingest SharePoint Online List through copy data etc. I found a guide towards File / Library. But not Normal list without the usage of ACS. I've done everything that is needed in…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Error code - 23353 Getting Error 403 Forbidden while using the copy activity
I am using the copy activity and when i use the mapping section then this showing the error - Rest call failed with client error, status code 403 Forbidden, please check your activity settings. Request URL:…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Moving SHIR binaries from C: to D: drive
Hi, We have installed Microsoft Selfhosted Integration Runtime(SHIR (both Azure and On‑Premises) on a virtual machine, where the default installation is on the C: drive. We came to know that there is a possibility to move the SHIR binaries from C: to…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Invalid linked service properties provided. The length of argument 'password' exceeds its limit of '128'.
Invalid linked service properties provided. The length of argument 'password' exceeds its limit of '128'.
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Adf Pipeline created date
Hi Team, How can we find the excat adf pipeline created date. I tried to get detils from Activity log but i can see ony 3 months data can be fetched. Please suggested how can i get the pipeline creation date. Regards Saim
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
What are the supported Open SSH Key Formats when it is using in Azure ADF - Linked services for SFTP access
Hi, I created a ADF Linked services with Open SSH key for SFTP access. It is supports only with RSA format key but not ECDSA format. As per the security requirement, I need to use only ECDSA format. Please help me to provide the suggestions and…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Connection to UiPath with Service Principal in Azure Data Factory
Dear All, I'm very new to Azure Data Factory and recently I got a request to connect to UiPath using Service Principal. I have received Directory (tenant) ID, Application (client) ID, Client secret Value and Secret ID. I could also see that it is already…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
ADF - Az function activity 499 error
WE get an error while executing azure function activity inside adf pipeline. Azure function is calling an API and extracts and stores the data returned by api to a storage account. Call to provided Azure function 'api_blob' failed with status-'499'…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Fix format of a datetime column in Copy Activity's Mapping section
We have set format of a Datetime column to "G" and "en-US" in mapping section of copy activity. Here our source is Azure SQL db and Destination is ADLS. The format of the datatype is as expected. Here is the type conversion setting:…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Can the Azure SQL database be scaled up (in the DTU model) automatically before running a pipeline and then scaled down afterwards?
I have several pipelines in ADF that when executed write data to an azure sql database. I am using the Standard DTU model for the pricing tier. Using any available ADF activities, can the Azure SQL database be scaled up (in the DTU model) automatically…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
How to Call a ORACLE DATABASE PACKAGE from Azure data factory and receive the output of that package.
Team I need to call a ORACLE DATABASE PACKAGE from azure data factory and use its output in other activities but i am not able to call a PACKAGE. Inside the package there are many stored procedures written so i need to call a specific stored procedure…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
I have to copy data from share point (Unstructured Data Photo/Video)to Azure blob storage using ADF pipeline
Can anyone tell me which activities I have to use as well which access I have required on share point level as well On Entra ID As well which linked service I have to use
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Column level type conversion in Mapping section of Copy Activity in Azure Data Factory.
Earlier, I used to get the option to change type conversion settings of a particular datetime column in mapping section of a copy data activity. Here my source is Azure SQL DB and Destination is Azure blob storage. Now, If I am changing my source to…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.