Share via

Columns disappear when I extract data from dataverse using synapse

Arquitectura Datos 0 Reputation points
2024-09-06T16:55:18.29+00:00

I am trying to extract data from dataverse source, but when I execute the pipeline, the output file does not contain some columns, I thought it was my fetchXML query but when I executed it using external tools the columns are shown, so :

  • It is not the FetchXML Query
  • The columns exist

What could be the reason?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,003 questions
Microsoft Dataverse Training
Microsoft Dataverse Training
Microsoft Dataverse: A Microsoft service that enables secure storage and management of data used by business apps. Previously known as Common Data Service.Training: Instruction to develop new skills.
37 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Amira Bedhiafi 26,491 Reputation points
    2024-09-07T15:41:06.88+00:00

    Check if the Synapse pipeline has any transformations or mappings that are altering the schema. Sometimes during data ingestion, pipelines might drop or rename columns if they are not properly configured.

    If the schema inference does not detect all the columns or there is a mismatch between data types, it may omit some columns. You can try manually specifying the schema or using schema mapping in your dataflow.

    Data masking or security policies in Dataverse could also prevent some columns from being extracted under certain conditions. Ensure that there are no row-level security (RLS) or column-level security (CLS) policies in place that restrict certain columns from being visible during extraction.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.