Is Azure Data Factory planning to add support for ztsd compression on parquet files?

Question

Is Azure Data Factory planning to add support for ztsd compression on parquet files?

Gabriel Eckers 0

I've run into some issues previewing parquet data using a pipeline Copy Activity when that data was compressed using the zstd codec. Are there plans for ADF to add support for this (now default in Databricks) compression method?

Example error:

An error occurred when invoking java, message: java.lang.UnsatisfiedLinkError:D:\Users\_azbatchtask_2\AppData\Local\Temp\libzstd-jni-1.5.5-51665259375217484100.dll: Your organization used Device Guard to block this app. Contact your support person for more info
no zstd-jni-1.5.5-5 in java.library.path
Unsupported OS/arch, cannot find /win/amd64/libzstd-jni-1.5.5-5.dll or load zstd-jni-1.5.5-5 from system libraries. Please try building from source the jar or providing libzstd-jni-1.5.5-5 in your system.
total entry:30
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864)
java.lang.Runtime.loadLibrary0(Runtime.java:870)
java.lang.System.loadLibrary(System.java:1122)
com.github.luben.zstd.util.Native$1.run(Native.java:69)
com.github.luben.zstd.util.Native$1.run(Native.java:67)
java.security.AccessController.doPrivileged(Native Method)
com.github.luben.zstd.util.Native.loadLibrary(Native.java:67)
com.github.luben.zstd.util.Native.load(Native.java:154)
com.github.luben.zstd.util.Native.load(Native.java:85)
com.github.luben.zstd.ZstdOutputStreamNoFinalizer.<clinit>(ZstdOutputStreamNoFinalizer.java:18)
com.github.luben.zstd.RecyclingBufferPool.<clinit>(RecyclingBufferPool.java:18)
org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:90)
org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:83)
org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:112)
org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader.readDictionaryPage(ColumnChunkPageReadStore.java:236)
org.apache.parquet.column.impl.ColumnReaderBase.<init>(ColumnReaderBase.java:410)
org.apache.parquet.column.impl.ColumnReaderImpl.<init>(ColumnReaderImpl.java:46)
org.apache.parquet.column.impl.ColumnReadStoreImpl.getColumnReader(ColumnReadStoreImpl.java:82)
org.apache.parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:271)
org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:147)
org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:109)
org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:177)
org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:109)
org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:141)
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:230)
org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:132)
org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
com.microsoft.datatransfer.bridge.parquet.ParquetBatchReaderBridge.<init>(ParquetBatchReaderBridge.java:70)
com.microsoft.datatransfer.bridge.parquet.ParquetBatchReaderBridge.open(ParquetBatchReaderBridge.java:64)
com.microsoft.datatransfer.bridge.parquet.ParquetFileBridge.createReader(ParquetFileBridge.java:22)
.

Gabriel Eckers 0 Reputation points

2025-11-21T15:00:07.3666667+00:00

Additional context, this is NOT using a self hosted integration runtime.
VRISHABHANATH PATIL 1,725 Reputation points Microsoft External Staff Moderator

2025-11-28T08:41:27.45+00:00

Hi @Gabriel Eckers

I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
VRISHABHANATH PATIL 1,725 Reputation points Microsoft External Staff Moderator

2025-11-29T03:46:41.01+00:00

Hi @Gabriel Eckers

Just checking in to see if you had any questions. We are happy to assist and continue the discussion whenever you are ready.
Gabriel Eckers 0 Reputation points

2025-11-29T20:36:17.86+00:00

Thanks for following up.

Unfortunately, the AI-generated answer didn't address my question. I know that zstd compression is not currently supported in ADF. I'm looking to find out if it's on the roadmap.

If not, I'll have to investigate alternative tooling to handle this use case.
VRISHABHANATH PATIL 1,725 Reputation points Microsoft External Staff Moderator

2025-11-30T05:46:43.27+00:00

Hi @Gabriel Eckers,**
**
Thank you for sharing your feedback. We have included a reference KB article link in the answer section for your convenience. Kindly review it to help resolve your issue.

2 answers

Your answer

Gabriel Eckers 0 Reputation points

2025-11-21T15:00:07.3666667+00:00

Additional context, this is NOT using a self hosted integration runtime.
VRISHABHANATH PATIL 1,725 Reputation points Microsoft External Staff Moderator

2025-11-28T08:41:27.45+00:00

Hi @Gabriel Eckers

I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
VRISHABHANATH PATIL 1,725 Reputation points Microsoft External Staff Moderator

2025-11-29T03:46:41.01+00:00

Hi @Gabriel Eckers

Just checking in to see if you had any questions. We are happy to assist and continue the discussion whenever you are ready.
Gabriel Eckers 0 Reputation points

2025-11-29T20:36:17.86+00:00

Thanks for following up.

Unfortunately, the AI-generated answer didn't address my question. I know that zstd compression is not currently supported in ADF. I'm looking to find out if it's on the roadmap.

If not, I'll have to investigate alternative tooling to handle this use case.
VRISHABHANATH PATIL 1,725 Reputation points Microsoft External Staff Moderator

2025-11-30T05:46:43.27+00:00

Hi @Gabriel Eckers,**
**
Thank you for sharing your feedback. We have included a reference KB article link in the answer section for your convenience. Kindly review it to help resolve your issue.

Answer 1

Currently, Azure Data Factory (ADF) does not support the Zstandard (zstd) compression codec for Parquet files. The supported compression codecs for Parquet files include options like none, gzip, snappy, and lzo (though lzo is not supported for read/write operations in Copy activity). If you are encountering issues with zstd compression, it may be necessary to consider alternative compression methods that are supported by ADF until zstd support is officially added in the future.

For the error you are experiencing, it appears to be related to the inability of the Java environment to load the zstd native library. This could be due to security settings such as Device Guard blocking the library from being loaded. You may want to consult your IT support to address this issue regarding the library loading or consider using a different compression codec that is supported by ADF.

References:

Answer 2

VRISHABHANATH PATIL 1,725 Microsoft External Staff Moderator

Hi @Gabriel Eckers

Public ADF documentation still does not list ZSTD among supported Parquet codecs for classic ADF pipelines. No official doc indicates near-term ZSTD support in classic ADF.

https://learn.microsoft.com/en-us/azure/data-factory/format-parquet

Share via

Is Azure Data Factory planning to add support for ztsd compression on parquet files?

2 answers

Your answer