Share via

copy tables from sql server on-prem to .parquet format in delta lake

Zaeemul Haque 45 Reputation points
2024-10-29T19:03:53.5233333+00:00

Currently our copy process for 10 tables is running in pipeline in ADF, our source is ON-PREM SQL Server and target is in BLOB Storage GEN2.

it copies data(tables) in parquet file format in HOT storage Account --- StorageV2 (general purpose v2) but I need to change the target as delta lake praquet file format, target will be remain the same Hot Storage ---- How can it be done.

I checked the ms document --- https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-databricks-delta-lake?tabs=data-factory

it seems to be similar but not what I am looking

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,233 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,221 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,864 questions
Accepted answer
  1. Chandra Boorla 2,990 Reputation points Microsoft Vendor
    2024-10-30T00:05:23.8233333+00:00

    Hi @Zaeemul Haque

    Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

    As I understand that you want to modify your existing Azure Data Factory pipeline to copy data from your on-premises SQL Server to Delta Lake format in your Blob Storage Gen2 (Hot storage). While ADF doesn't directly handle this conversion, we can achieve it using Azure Synapse Notebooks.

    Here is the step-by-step procedure to achieve your requirement.

    Maintain Existing Copy Activity configuration (Parquet Format): Create a pipeline in Azure Synapse Analytics that copies data from the on-premises SQL Server to Parquet format in your Blob Storage.

    Add Synapse Notebook Activity for Conversion: After the copy activity, add a new Synapse Notebook activity. This notebook will handle the Parquet to Delta Lake conversion.

    Create Conversion Code in Synapse Notebook: The notebook will use Spark SQL to read the Parquet files and write them as Delta Lake tables. Here is the conversion code.

    User's image Note: Please replace your storage details and file paths.

    I have tried to repro the scenario, please refer to the gif file for more details. DeltaLake

    I hope this information helps. Please do let us know if you have any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.