Azure Databricks

1 answer

Query on Using Azure Free Account for Delta Live Tables (DLT) POC

We are currently exploring the latest features of Databricks Delta Live Tables (DLT) and are planning to build a small proof of concept (POC) to evaluate these capabilities. We request your guidance on the following: Free Azure Account Usage: Can we…

asked

Janice Chi 140

accepted

Janice Chi 140

0 answers

How to selectively re-run only failed ETL phases (extract/transform/load) for specific partitions in a continuous pipeline?

Question: In our current reconciliation framework, we log the success/failure of each ETL pipeline phase—extraction, transformation, and load—using dedicated status flags (extraction_status, transformation_status, load_status) in a phase-level status…

asked

Janice Chi 140

0 answers

Best Retry Strategy for Reconciliation-Failed Microbatches in Real-Time CDC Pipelines (Azure SQL + Databricks)

We are implementing a near-real-time CDC streaming pipeline using Azure Databricks and Azure SQL Hyperscale. Flattened CDC data from Kafka is ingested into Delta Lake (ADLS) per microbatch, and each batch is also written to a staging table in Azure SQL.…

asked

Janice Chi 140

commented

Janice Chi 140

1 answer

When will SQL Warehouse Serverless be available in South Africa North Region?

I am interested in using the Serverless SQL Warehouse cluster for my projects. Could you please provide information on when this will be available in the South Africa North region?

asked

Tabina Shabir 20

accepted

Tabina Shabir 20

3 answers

How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute

Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…

asked

Krzysztof Przysowa 20

answered

Yunpeng Tang 0 Microsoft Employee

0 answers

The workspace is in a failed state and cannot be launched.

When I try to delete a Databricks resource - I get the error... The workspace 'workspace_name' is in a failed state and cannot be launched. Please review error details in the activity log tab and retry your operation. How do I resolve this? I've…

asked

Sukhminder Bining 0

commented

Sukhminder Bining 0

0 answers

Not able to create the CREATE STORAGE CREDENTIAL as storage credential is not recognized command in azure databricks

I am trying to create the external location using the command . when I run the command then it does not recognize the storage credential then i can't run the external location command also. please help so that i can run the migrate table command in…

asked

padam 0

commented

Smaran Thoomu 24,260 Microsoft External Staff Moderator

0 answers

Databricks Issue to use Lakebase / OLTP

Hello, We try to use new feature of Databricks regarding OLTP. we're having a problem. By following this documentation and doing the setup from the UI. https://learn.microsoft.com/en-us/azure/databricks/oltp/create/ …

asked

Maxime DEBOFFLE 0

commented

Smaran Thoomu 24,260 Microsoft External Staff Moderator

1 answer

Not able to login to account level at azure databricks

I am working on Unity Catalog migration from hive metastore using UCX migration and I am not able to login to account level i.e. https://accounts.azuredatabricks.net. it redirects to workspace url after login . I need to go to account level login so that…

asked

padam 0

commented

Smaran Thoomu 24,260 Microsoft External Staff Moderator

1 answer

Cursor position issue in Azure Databricks

Hi, when editing code inside Databricks notebook environment, the cursor does not position itself at where I thought I was typing. There seems to be difference in 5-6 character positions between where the cursor is and where characters are…

asked

Hong Ling TAN (NP) 20

commented

Venkat Reddy Navari 3,060 Microsoft External Staff Moderator

1 answer

Guidance on Real-Time Streaming with Reconciliation and Domain Cutover Handling – Kafka → Databricks → Azure SQL Hyperscale

We are currently designing a real-time ingestion and reconciliation architecture using Databricks Structured Streaming that reads IBM DB2 CDC messages via Kafka and writes them to Azure SQL Hyperscale. The context is a healthcare data migration…

asked

Janice Chi 140

commented

Venkat Reddy Navari 3,060 Microsoft External Staff Moderator

1 answer

Using Dynamic Job Parameters with Databricks Job Activity

A replacement is being implemented for a generic Databricks job trigger pipeline with the new Databricks job activity. The goal is to maintain all job triggering logic in a separate pipeline. Currently, a JSON-like string is being dynamically passed with…

asked

Víctor C. Fernández 0

commented

Venkat Reddy Navari 3,060 Microsoft External Staff Moderator

1 answer

single parameterized notebook for all layer-wise row count and hash reconciliation

Full Question: In our Azure-based data migration project, we are implementing row count and hash reconciliation between different storage layers across two phases: One-time migration (DB2 Snapshot → ADLS CSV/Parquet → Azure SQL Hyperscale) Catch-up…

asked

Janice Chi 140

commented

Venkat Reddy Navari 3,060 Microsoft External Staff Moderator

1 answer

How to efficiently unify batch and streaming data ingestion using Azure Databricks and Delta Lake?

Hello Team, I'm building a pipeline that ingests both batch (via Data Factory) and streaming data (via Event Hubs) into Delta Lake on ADLS Gen2 using Azure Databricks. I'm following the medallion architecture (Bronze, Silver, Gold). What are the best…

asked

Ayoshna Saha 135

accepted

Ayoshna Saha 135

1 answer

Databricks internal claude the model has been reporting errors today

the error is: {"error_code":"TEMPORARILY_UNAVAILABLE","message":"The server is currently unable to handle the request."}, is azure updating this service?

asked

davaid 40

accepted

davaid 40

1 answer

does Claude 4 in databricks supports reasoning?

i found claude 4 sonnet in azure databricks, but don't know how to enable thinking for this model

asked

davaid 40

accepted

davaid 40

1 answer

Can We View Last Month's DBU Usage by Job or Cluster in Azure Portal or Databricks UI

Can we view the total DBU (Databricks Unit) usage for the last month, broken down by cluster or job, directly in the Azure Portal or within the Azure Databricks UI (such as under Compute or any built-in usage section)? If not, what is the recommended…

asked

Sudhakar P 145

commented

Chandra Boorla 14,585 Microsoft External Staff Moderator

1 answer

I want to create an Azure Databricks in which I can connect to ADLS gen 2. Can anyone please help me as I am unable to do so or if anyone can provide steps end to end.

asked

Sanyog Vaidya 0

commented

Chandra Boorla 14,585 Microsoft External Staff Moderator

1 answer

Request for Merge Strategy Guidance in Databricks for CDC Payload from Kafka (IBM InfoSphere CDC)

We are currently designing a large-scale Change Data Capture (CDC) ingestion pipeline in a healthcare data platform, and we would appreciate your guidance regarding merge strategy performance and payload optimizations in Azure Databricks. 📌 Project…

asked

Janice Chi 140

commented

Chandra Boorla 14,585 Microsoft External Staff Moderator

1 answer

Unable to attach databricks workspace to unity catalog

Unable to attach data bricks workspace to unity catalog using my personel azure account

asked

Rishi Anand 0

commented

Venkat Reddy Navari 3,060 Microsoft External Staff Moderator

Azure Databricks

Filter

Content

2,521 questions with Azure Databricks tags

Query on Using Azure Free Account for Delta Live Tables (DLT) POC

How to selectively re-run only failed ETL phases (extract/transform/load) for specific partitions in a continuous pipeline?

Best Retry Strategy for Reconciliation-Failed Microbatches in Real-Time CDC Pipelines (Azure SQL + Databricks)

When will SQL Warehouse Serverless be available in South Africa North Region?

How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute

The workspace is in a failed state and cannot be launched.

Not able to create the CREATE STORAGE CREDENTIAL as storage credential is not recognized command in azure databricks

Databricks Issue to use Lakebase / OLTP

Not able to login to account level at azure databricks

Cursor position issue in Azure Databricks

Guidance on Real-Time Streaming with Reconciliation and Domain Cutover Handling – Kafka → Databricks → Azure SQL Hyperscale

Using Dynamic Job Parameters with Databricks Job Activity

single parameterized notebook for all layer-wise row count and hash reconciliation

How to efficiently unify batch and streaming data ingestion using Azure Databricks and Delta Lake?

Databricks internal claude the model has been reporting errors today

does Claude 4 in databricks supports reasoning?

Can We View Last Month's DBU Usage by Job or Cluster in Azure Portal or Databricks UI