Azure ML v2 mode=upload output to an ADLS Gen1 datastore fails with "MSCONCAT 0x83090a70 (gen1-to-gen2 symlink)" for files larger than ~8 MB

Johnny Xiao 20 Reputation points Microsoft Employee
2026-07-30T05:18:46.6766667+00:00

I have an Azure Machine Learning v2 pipeline (azure-ai-ml SDK). A step writes an output file to an ADLS Gen1 datastore (datastore type AzureDataLakeGen1) using a uri_file output with mode=upload. My script writes the file to the local output mount and the Azure ML runtime uploads it.

A small output (~7.8 MB) uploads fine, but a larger output (~8.3 MB and up) fails during the upload finalize with:

MSCONCAT failed with error 0x83090a70 (Bad Request. The specified operation is not valid for gen1-to-gen2 symlink.)

The target folder was migrated from ADLS Gen1 to Gen2 and is exposed as a gen1-to-gen2 symlink, and the storage service rejects the WebHDFS MSCONCAT operation on it. The upload is done by the Azure ML common runtime (rslex), not by my code.

What I have confirmed:

  • It is not a permissions issue. The same job, with the same datastore identity, writes the smaller file to the same folder successfully. Only the larger file fails.
  • The difference seems to be a single-block upload threshold around 8 MiB. Below it, the file is written in one stream with no concat and succeeds. Above it, rslex splits into blocks and finalizes with MSCONCAT, which the migrated-symlink folder rejects.
  • Writing the same large file to a brand-new (non-migrated) folder works. Only the migrated gen1-to-gen2 symlink folders reject MSCONCAT.

What I tried, with no luck:

  • mode=rw_mount instead of upload: the flush also finalizes with MSCONCAT and fails the same way (shows up as OSError Errno 5).
  • Forcing the sequential ADLS Gen1 copier with the env var DATASET_ADLSGEN1_USE_ALTERNATIVE_COPIER=false. Setting it in the environment image had no effect (the uploader still ran the default). Setting it through the job's environment_variables did reach the uploader, but the write still failed with the same MSCONCAT error, so the sequential copier also uses MSCONCAT for multi-block uploads.

I cannot shrink the file below the threshold and I must keep writing to this ADLS Gen1 datastore.

Questions:

  1. Is there a supported way to make the managed ADLS Gen1 upload avoid the MSCONCAT finalize (write as a single stream or append-only) for files above the single-block threshold?
  2. Is the single-block/single-PUT size threshold configurable (for example an azureml-dataprep-rslex block-size setting) so a 10-50 MB file is written as one block with no concat, and can it be passed through the job's environment_variables?
  3. Is MSCONCAT on a gen1-to-gen2 symlink a known limitation, and is a runtime fix planned?
  4. What is the recommended pattern for writing files larger than ~8 MB from an Azure ML job to an ADLS Gen1 datastore whose folders were migrated to gen2 symlinks?

Environment: azure-ai-ml v2, command/pipeline job, mode=upload, uri_file output, datastore type AzureDataLakeGen1, Azure ML common runtime with azureml-dataprep-rslex ~2.25, Linux compute.

Azure Data Lake Storage
Azure Data Lake Storage

An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.

0 comments
Answer accepted by question author
Allan Solomon Mejia 8,085 Reputation points
2026-08-07T21:23:04.88+00:00

Hello @Johnny Xiao

Thank you for the detailed investigation. Based on the behavior you've described, this does not appear to be an issue with your pipeline code or permissions. Your testing strongly suggests the failure occurs during the Azure ML runtime's upload finalization when it issues the WebHDFS MSCONCAT operation.

The key observations are:

  • The same identity and datastore can upload smaller files successfully.
  • The failure occurs consistently once the file exceeds the apparent multi-part upload threshold (~8 MiB).
  • The same large file succeeds when written to a non-migrated ADLS Gen1 path.
  • Both mode=upload and mode=rw_mount fail during the same MSCONCAT finalize step.

This points to the interaction between the Azure ML runtime (azureml-dataprep-rslex) and ADLS Gen1 paths exposed as Gen1-to-Gen2 symlinks, rather than an authentication or authorization issue.

From the Azure ML documentation, there is no documented setting to disable MSCONCAT, force a single-stream upload above the internal threshold, or configure the block size used by the managed upload pipeline. Those implementation details are handled internally by the Azure ML runtime.

At this point, I'd recommend a Microsoft moderator engage the Azure Machine Learning engineering team to confirm whether:

  • MSCONCAT against Gen1-to-Gen2 symlinked folders is a known limitation.
  • The ~8 MiB threshold is fixed or configurable.
  • A newer version of azureml-dataprep-rslex changes this upload behavior.

To help with the investigation, could you also provide:

  1. The Azure ML SDK version (azure-ai-ml).
  2. The exact version of azureml-dataprep-rslex used by the compute environment.
  3. Whether the same behavior occurs on another compute target or environment image.
  4. Whether writing directly to an Azure Data Lake Storage Gen2 datastore (instead of the migrated Gen1 endpoint) succeeds for files larger than 8 MiB.

Since the upload is performed by the managed Azure ML runtime rather than your application code, this information will help determine whether the issue is a known runtime limitation or a regression.

Based on the evidence you've collected, this looks like a service/runtime behavior rather than something that can be worked around through SDK configuration. An engineering review would be the appropriate next step to determine whether this is a known limitation of MSCONCAT on migrated Gen1-to-Gen2 symlink paths or a defect in the Azure ML upload runtime.

Please "Accept the Answer" if this information helped you. This will help us and others in the community as well.

Was this answer helpful?

1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Johnny Xiao 20 Reputation points Microsoft Employee
    2026-08-07T22:10:00.5366667+00:00

    Thank you for the very thorough analysis. Your assessment is correct, and we have now

    confirmed the root cause with the ADLS/Cosmos storage-migration team that owns this account.

    ────────────────────────────────────────────────────────────────────────

    ROOT CAUSE (confirmed with the storage-migration team)

    ────────────────────────────────────────────────────────────────────────

    The failure is not in our pipeline code or permissions. It is a known limitation of the

    Gen1-to-Gen2 storage migration, exactly as you hypothesized:

    • The output folder used by our pipeline was migrated from ADLS Gen1

    to Gen2 earlier this year and now exists as a "gen1-to-gen2 symlink."

    • Migration eligibility is driven by telemetry: paths that show MSCONCAT (concat) usage are

    excluded / deferred, because concat is not supported on migrated Gen2 symlink paths. This

folder had only ever written small, single-stream payloads (below the multi-part upload

threshold), so its telemetry showed no concat usage — and it was therefore migrated.

    • Now that the model output exceeds the multi-part threshold (~8 MiB), the Azure ML runtime

    finalizes the upload with a WebHDFS MSCONCAT call, which the migrated symlink path rejects

(error 0x83090a70). The path is already migrated, so it cannot be reverted for this folder.

    This matches your conclusion precisely: it is the interaction between the Azure ML runtime's

    block-upload finalize (MSCONCAT) and a migrated Gen1-to-Gen2 symlink path — a storage-platform

    limitation, not an authentication/authorization problem and not our application code. An Azure ML

    engineering escalation is therefore likely not required; the storage team is (1) deferring

    migration for concat-using paths going forward and (2) building an interim concat solution

    (tentatively targeted around August 2026, per the storage team — not yet confirmed).

    ────────────────────────────────────────────────────────────────────────

    REQUESTED DETAILS

    ────────────────────────────────────────────────────────────────────────

    1. Azure ML SDK (azure-ai-ml): 1.31.0
    2. Compute-environment runtime packages (from the job's data-capability telemetry): azureml-dataprep-rslex: 2.25.4 azureml-dataprep: 5.4.3 azureml-core: 1.61.0.post3 common-runtime data-capability build: 0.0.1.20260615.1 (Linux)
    3. Same behavior on another compute target / environment image: We have not yet tested a different compute target. The failure reproduced consistently across multiple runs on the same AmlCompute cluster, and under BOTH mode=upload and mode=rw_mount, always at the MSCONCAT finalize step. We can run a cross-compute repro if it would help the investigation.
    4. Writing directly to an ADLS Gen2 datastore for files > 8 MiB: Yes — it succeeds. As a diagnostic we wrote a 26,214,400-byte (25 MiB) file directly to the Gen2 blob endpoint of the same account; it landed intact with no MSCONCAT / block-concat finalize step. (Caveat: the storage team advised against using the Gen2 container directly while the account is under migration — direct writes bypass the migration symlink, are not visible from the Cosmos/Gen1 view, and can cause naming conflicts — so we are treating this only as confirmation of the behavior, not as a production path yet.)

    ────────────────────────────────────────────────────────────────────────

    SUPPORTING EVIDENCE (from the runtime logs)

    ────────────────────────────────────────────────────────────────────────

    • Failing object: the model output (ONNX), 8,748,891 bytes (8.34 MiB).

    • WebHDFS call: POST <redacted-endpoint>/webhdfs/v1/<redacted-path>

                      ?op=MSCONCAT&deleteSourceDirectory=true&api-version=2018-09-01

                  → HTTP 400: "MSCONCAT failed with error 0x83090a70 (Bad Request. The

                  specified operation is not valid for gen1-to-gen2 symlink.)"

                  Endpoint: a Gen1 WebHDFS endpoint (*.azuredatalakestore.net).

    • Smaller models in the same pipeline (e.g., the ~7.8 MiB text-ads model) upload successfully

    with the same identity and datastore — a single-part upload with no MSCONCAT — which is

consistent with the ~8 MiB multi-part threshold you noted.

    ────────────────────────────────────────────────────────────────────────

    NEXT STEPS

    ────────────────────────────────────────────────────────────────────────

    Because the root cause and ownership sit with the storage-migration team, we are proceeding with

    one of their interim options: (a) write to a new, non-migrated Gen1 path (where MSCONCAT still

    works) until the concat solution ships, or (b) move the connector to a new Gen2 container fronted

    by a new Cosmos symlink. We're happy to keep this thread open if you'd still like Azure ML

    engineering to confirm whether the ~8 MiB multi-part threshold or the rslex 2.25.4 upload

    behavior is configurable.

    Thanks again for the careful triage.

    Was this answer helpful?

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.