Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,496 questions
1 answer
One of the answers was accepted by the question author.
tagging strategy implementation
Hello, when the Azure Data Lake design is done, partitioning and tagging are key performance factors as they ensure smooth transaction patterns. I am not quite clear on the tagging aspects, where and what all components should be covered, and what needs…
asked 2024-11-21T13:48:37.9033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 88%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azure_learner
420
Reputation points
accepted 2024-11-23T09:16:48.9+00:00
azure_learner
420
Reputation points
0 answers
Read AppServiceConsoleLogs and push to Blob Storage container/table
Hello Team, we have a webapp python Django based. We are generating custom logs for user actions using code below. Settings.py: LOGGING = { "version": 1, "disable_existing_loggers": False, "formatters": { …
asked 2024-11-20T09:31:14.9233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 35%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BalaKrishna Mahamkali
0
Reputation points
commented 2024-11-23T04:49:09.6133333+00:00
VenkateshDodda-MSFT
21,991
Reputation points Microsoft Employee
1 answer
Loading large nested JSON files into Azure SQL db using Azure Data Factory
Hi all, I am currently working with large nested JSON files stored inside a blob container. Each file has a different structure with varying fields and data types, and nested arrays. My goal is to load these files into an Azure SQL database in an…
asked 2024-11-22T11:00:34.1533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 17%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
A K
45
Reputation points
answered 2024-11-22T19:33:25.5633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sina Salam
12,976
Reputation points
0 answers
Can't write to blob storage from AzureML Spark Cluster
Using the Azure ML Spark compute (serverless or attached), it is not possible to write to gen2 datalake blob storage. The code below produces the error 'Caused by: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException:…
asked 2023-05-10T18:32:20.2533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 40%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDC%3C/text%3E%3C/svg%3E)
Dance, Cody R. (ALT)
35
Reputation points
commented 2024-11-22T13:10:06.4866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 91%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)
thomas
1
Reputation point
1 answer
How to load multiple different JSON files into Azure SQL DB using Azure Data Factory
Hi all, I am currently working with some nested JSON files and attempting to load it from the blob container to azure sql db using azure data factory. All of the JSON files have different structures as the fields and columns vary in each one, which is…
asked 2024-11-22T10:42:29.4266667+00:00
A K
45
Reputation points
answered 2024-11-22T11:25:59.1833333+00:00
Nandan Hegde
32,656
Reputation points MVP
2 answers
Data Share support ADLS Gen2 with private endpoint
Do data share support ADLS Gen2 that is deployed with Private Endpoint?
asked 2020-09-30T16:38:02.05+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 13%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAT%3C/text%3E%3C/svg%3E)
Alvin Tai
1
Reputation point Microsoft Employee
commented 2024-11-21T09:06:00.5333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 64%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIC%3C/text%3E%3C/svg%3E)
Ian Carrick (csu)
0
Reputation points
1 answer
Azure ML | Support for ADLS Gen2 Datastore with SAS Token Authentication
Hello, I’m currently working on an application where we’re connecting various data sources—such as file shares and ADLS Gen2—to Azure Machine Learning. While we can create file-share datastores with SAS token authentication, I noticed that this option…
asked 2024-11-15T08:37:53.9266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 41%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPL%3C/text%3E%3C/svg%3E)
Pål Hannus
0
Reputation points
edited a comment 2024-11-20T08:28:22.0366667+00:00
Pål Hannus
0
Reputation points
1 answer
Data Quality issue in the Purview
Hello Team, We have configured the ADLS2 as source and scan it. For the data Quality, we did the following the steps : Create the Governance domain and publish it. Create the Data Product and add the tables. In the Data Quality section, add the…
asked 2024-11-11T15:53:43.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 23%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDP%3C/text%3E%3C/svg%3E)
Dhaval Pujara
0
Reputation points
edited a comment 2024-11-19T01:03:02.2+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
3,535
Reputation points Microsoft Vendor
1 answer
Getting "storage_root does not specify a URI scheme." while setting ADLS path in default metastore.
Hi Team, I am facing while setting the ADLS path for the existing default metastore in account console page in azure. Below is the image showing i have already created the container and have enabled Hierarchical namespace as well. But getting below…
asked 2024-11-18T11:01:44.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 53%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Akancha
0
Reputation points
answered 2024-11-18T17:17:49.2133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 6%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKD%3C/text%3E%3C/svg%3E)
Keshavulu Dasari
1,830
Reputation points Microsoft Vendor
2 answers
Failure happened on 'destination' side. ErrorCode=DeltaInvalidCharacterInColumnName.
This exception occurred when I use the pipeline of data factory to copy data from sql server to lakehouse. But I didn't find any problems with the raw data.
asked 2024-03-18T13:42:24.5+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 24%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EXY%3C/text%3E%3C/svg%3E)
Xavier Yan
0
Reputation points
commented 2024-11-14T15:48:33.4933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 67%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
asktocedsq
20
Reputation points
1 answer
One of the answers was accepted by the question author.
error code 2011
i am testing a pipeline, i introduced a repeated row in one of the files that i want to upload. i was expecting that the pipeline would have run anyway, uploading the correct files and not the incorrect one. actually, the entire pipeline did not work...…
asked 2024-11-11T16:28:19.3633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 83%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGE%3C/text%3E%3C/svg%3E)
Galvan, Elena (Bip Group)
20
Reputation points
commented 2024-11-14T07:31:57.2466667+00:00
Galvan, Elena (Bip Group)
20
Reputation points
1 answer
In MS fabric users able to view data but cannot download it
In MS Fabric, I want users to be able to view data in the workspace but not be able to download it. Please provide clear steps, along with links to verify the authenticity of the solution provided.
asked 2024-11-11T11:51:04.92+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 11%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Jain, Shreyans
0
Reputation points
commented 2024-11-13T18:06:00.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
1,510
Reputation points Microsoft Vendor
1 answer
Copy Files from sharepoint online site to azure datalake storage
Hello We are trying to setup the flow which will copy files from sharepoint online site to azure datalake storage. As per my understanding there are 2 options : Using ADF to pull the files as mentioned in the link below…
asked 2024-11-06T22:12:34.5+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 35%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sourav
105
Reputation points
edited the question 2024-11-13T08:57:32.6533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 17%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGM%3C/text%3E%3C/svg%3E)
Givary-MSFT
33,706
Reputation points Microsoft Employee
1 answer
504.0 GatewayTimeout & Invoking Azure function failed with HttpStatusCode - 499.
We've developed an Azure Function in python that connect to a Blob Storage, reads files and writes into in Azure tables. During the process, using Azure Functions & it's running fine for small size files (Less than 100 MB). The problem is that, when…
asked 2024-11-12T09:44:30.44+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 67%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
syamprasad kari
0
Reputation points
answered 2024-11-12T13:21:53.56+00:00
Amira Bedhiafi
26,971
Reputation points
1 answer
Alternative Methods for Capturing Data Lake Size in Less Time
Need assistance in capturing the size of the data lake per environment (e.g., Dev, SIT, Prod). Currently, a PowerShell script is used to fetch details, generating a CSV file for each environment with the medallion, folder, subfolder, and size. The…
asked 2024-11-11T15:45:58.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 8%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Arianne Chung
0
Reputation points
answered 2024-11-11T19:39:38.5766667+00:00
Keshavulu Dasari
1,830
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
Transforming JSON files using data flow
Hello! I currently have about 60 json files inside a blob container which most of them have different fields and values. I have created a pipeline with a get metadata activity that points to the container, with the field list set to Child items. I have…
asked 2024-11-07T15:23:15.93+00:00
A K
45
Reputation points
commented 2024-11-11T19:05:56.01+00:00
Keshavulu Dasari
1,830
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Data lake solutions
We are in the process of Data Lake and going further down the line we are really getting confused whether to go for delta lake , datalakehouse, or synapse analytics. The subtle nuances making things not easier such as " A Data Lake House merges…
asked 2024-11-08T17:01:54.56+00:00
azure_learner
420
Reputation points
accepted 2024-11-11T14:31:57.8033333+00:00
azure_learner
420
Reputation points
1 answer
One of the answers was accepted by the question author.
Why function could not find file
Hi there, I built an Azure Function to process json data from external requests, and then saved the json to a local file, uploaded it to the Container through the storage client. It worked well locally, and once deployed to Azure, it would prompt that…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 53%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZ%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
Data lake schema enforcement
Hello, In Data Lake data is processed or ingested as schema on read and that is data is read in it format that it comes from the source. But I read an article that says schema enforcement makes data lakes high-performance and data readable. Please…
asked 2024-11-03T14:25:43.58+00:00
azure_learner
420
Reputation points
accepted 2024-11-08T17:03:00.9133333+00:00
azure_learner
420
Reputation points
4 answers
How to send a mail notification for a failed pipeline in Azure Synapse Analytics?
How can I send a notification email to a specific email address without using a logic app when one of my Synapse Analytics pipelines fails? I would like to include the error message in the email notification.
asked 2024-03-20T15:48:58.53+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 69%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Aditya Singh
160
Reputation points
answered 2024-11-07T21:37:08.3566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 15%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMD%3C/text%3E%3C/svg%3E)
Manuel Dones
0
Reputation points