Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,240 questions
0 answers
Create Materialized Views via DLT Pipelines
I am attempting to create materialized views via DLT pipelines in the "abc" catalog and "gold" schema based on different catalog and schema ("fmv"), which contains 20 tables. Workspace binding , NSG connections and access…
asked 2024-11-22T21:22:51.9733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 77%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
130
Reputation points
commented 2024-11-23T17:56:58.99+00:00
AnnuKumari-MSFT
33,556
Reputation points Microsoft Employee
1 answer
Unable to launch my databricks workspace after i started a new pay as u go subscription
Hello Guys, I had previously used pay as u go during 1 free month as i had credits then, after 15 it expired. Now today i have started a new pay as u go subscription and was trying to launch databricks workspace just by filling these basic details and…
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
1,024 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,240 questions
asked 2024-11-22T16:04:48.6833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 42%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mantri Sai Ashish
0
Reputation points
answered 2024-11-23T10:20:39.57+00:00
Vinodh247
24,091
Reputation points MVP
1 answer
Databrick pyspark partition query
I am writing login in azure databrick Pyspark. Please see attached input data and expected output. Id will be used as a partition For every id we need to choose one record with latest date but if any id ( e.g Id = 3) contain multiple record with same…
asked 2024-11-23T04:47:09.25+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 35%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
Pankaj Joshi
331
Reputation points
answered 2024-11-23T10:15:53.31+00:00
Vinodh247
24,091
Reputation points MVP
1 answer
ADF Azure Databricks Linked Service job cluster using policyId that specifies pool id
Hi, how can I set up an Azure Databricks linked service using a job cluster, using a job policy that specifies the driver and worker pool ids? In the linked service definition, I have selected "New job cluster", which requires me to supply a…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 73%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJL%3C/text%3E%3C/svg%3E)
0 answers
Read AppServiceConsoleLogs and push to Blob Storage container/table
Hello Team, we have a webapp python Django based. We are generating custom logs for user actions using code below. Settings.py: LOGGING = { "version": 1, "disable_existing_loggers": False, "formatters": { …
asked 2024-11-20T09:31:14.9233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 35%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BalaKrishna Mahamkali
0
Reputation points
commented 2024-11-23T04:49:09.6133333+00:00
VenkateshDodda-MSFT
21,991
Reputation points Microsoft Employee
1 answer
Still being billed for Azure after I cancelled it
I signed up for the free Azure and then I cancelled the subscription, but I'm still receiving bills. How do I go about removing everything off my name?
asked 2024-10-10T05:26:01.63+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 93%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDJ%3C/text%3E%3C/svg%3E)
Desire Joffe
0
Reputation points
commented 2024-11-22T18:55:42.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 20%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Pranay Reddy Madireddy
700
Reputation points Microsoft Vendor
1 answer
Integrate storage, database, webapp, and databricks into one VNET
We are hardening our ML platform in Azure. Basically, we want to put all the Azure services ( ADLS, PostgreSQL DB, Web App, and Databricks cluster) in one virtural network so all the traffic will be inside the Azure network whenever possible (except the…
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
1,024 questions
Azure Virtual Network
Azure Virtual Network
An Azure networking service that is used to provision private networks and optionally to connect to on-premises datacenters.
2,534 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,240 questions
Azure Web Application Firewall
Azure Web Application Firewall
An Azure service that provides protection for web apps.
330 questions
Azure Database for PostgreSQL
Azure Database for PostgreSQL
An Azure managed PostgreSQL database service for app development and deployment.
1,119 questions
asked 2024-11-22T17:43:49.06+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 27%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
P, John
200
Reputation points
edited an answer 2024-11-22T18:12:28.5266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 99%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDZ%3C/text%3E%3C/svg%3E)
Delano Zuurman
0
Reputation points
1 answer
Failure starting repl. Try detaching and re-attaching the notebook.
I am using Azure Databricks for a couple of months already. I didn't change any code or configuration recently but all notebook are failed to run. The error is : Failure starting repl. Try detaching and re-attaching the notebook. It failed for simple…
asked 2024-11-20T13:22:33.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 90%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Mark Lui
0
Reputation points
commented 2024-11-22T11:33:44.0633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sina Salam
12,976
Reputation points
1 answer
Spark config through Databricks
Hi, I am trying to access storage account through databricks (thru notebook) with storage access key. I keep encountering error when I do spark.config.set spark.conf.set( …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 36%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
1 answer
How to extract 7z file on azure databricks
How to extract 7z file on azure databricks..
asked 2022-10-04T06:33:14.607+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 91%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
adf575076
1
Reputation point
commented 2024-11-22T08:18:26.0566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 69%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETA%3C/text%3E%3C/svg%3E)
Tanjil Amin
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Auto loader
Hi, How to add new column automatically when it is not present in delta table
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
1 answer
Managing PCI-DSS Compliance and Access to Serverless Features in Azure Databricks
Hello Azure Community, I am currently using Azure Databricks with PCI-DSS compliance enabled in our workspace, as maintaining stringent security standards is crucial for our organization. However, I've discovered that once PCI-DSS compliance is turned…
asked 2024-11-19T13:37:14.5166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 68%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZA%3C/text%3E%3C/svg%3E)
Zohaib Altaf
0
Reputation points
commented 2024-11-22T01:35:23.5633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
3,535
Reputation points Microsoft Vendor
0 answers
I am not able to access the delta path after creating the delta table in the Azure databricks.
I am using Pay-as-you-go subscription and I am working on Azure databricks. In Databricks, when Delta table crated , we could see the storage location from the catalog explorer. But, when trying to see the details by accessing path, I am coming across…
asked 2024-11-21T10:56:01.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 57.99999999999999%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Sudarsan Akula
0
Reputation points
commented 2024-11-21T20:07:10.8133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
1,510
Reputation points Microsoft Vendor
1 answer
Init scripts stored in volumes are not getting executed via Job Shared cluster via ADF
Hello Team, We have stored our init script in Azure databricks managed volumes path and we want to execute notebook via Job Shared cluster, the pipeline we are triggering is via ADF but the job cluster gives an error as We are using Assigned Manged…
asked 2024-11-18T11:49:41.7666667+00:00
Ashwini Gaikwad
130
Reputation points
commented 2024-11-21T07:39:13.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
17,525
Reputation points Microsoft Vendor
1 answer
How to Spark Stream into azure PostgreSQL
I am trying spark stream data into postgresql via databricks, with delta tables and change data feed enabled. As far as I understand I can use JDBC or the databricks connector as seen here:…
asked 2024-11-19T16:52:06.9+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 21%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAL%3C/text%3E%3C/svg%3E)
Arik Levy
20
Reputation points
commented 2024-11-20T18:58:46.6633333+00:00
Sina Salam
12,976
Reputation points
1 answer
Error with Create Table USING DELTA LOCATION in training exercise
In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,240 questions
Azure Training
Azure Training
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Training: Instruction to develop new skills.
1,770 questions
asked 2024-05-01T13:00:09.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 31%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
James Mitchell
5
Reputation points
commented 2024-11-20T15:43:17.59+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 67%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Sylvain MENEUST
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Resolving GPU memory errors in databricks with PyTorch on g4dn.2xlarge instances
I'm encountering persistent memory issues while training and testing a PyTorch model on Databricks. In my Databricks job configuration, I’ve specified node_type_id and driver_node_type_id as g4dn.2xlarge. According to the documentation, this instance…
asked 2024-11-18T07:29:13.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 15%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG2%3C/text%3E%3C/svg%3E)
Gabriel-2005
365
Reputation points
accepted 2024-11-20T11:42:59.3566667+00:00
Gabriel-2005
365
Reputation points
1 answer
Table Access control option is not available in Azure Databricks Compute cluster
Hi, I am trying to enable Table Access control in one of my Azure Databricks Compute Cluster, the cluster was created around 02 years back with pricing tier selected as Standard, now we have upgraded the same cluster to Premium tier in order to…
asked 2024-11-15T15:16:40.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 24%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Satish Kumar
51
Reputation points
commented 2024-11-20T07:47:33.7033333+00:00
Ganesh Gurram
1,510
Reputation points Microsoft Vendor
1 answer
How to downgrade from TLS 1.3 to 1.2 in Azure Databricks env
Hi, As part of troubleshooting a current intermittent issue with connecting to an external API, I am being asked to downgrade the TLS version from 1.3 to 1.2. I am running a python notebook in our Azure Databricks environment and believe that TLS 1.3 is…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 50%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E_%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
How can i migrate jobs from one workspace to other workspace (Dev to QA)
How can i migrate jobs from one workspace to other workspace (Dev to QA) .
asked 2024-11-15T05:44:35.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
110
Reputation points
accepted 2024-11-20T05:09:57.46+00:00
Lakshmi Narayana Sarma Bhamidipati
110
Reputation points