Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
1 answer
Connecting to an HDInsight Kafka Cluster from a Local Machine via Python Script
Hello, I've configured my Kafka cluster on Azure HDInsight with a virtual network (VNet) that includes a NAT gateway and a Network Security Group (NSG). Now, I’m trying to connect to this cluster from my local Windows machine using a Python script to…
Azure HDInsight
asked 2025-07-01T12:55:51.3266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 93%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHO%3C/text%3E%3C/svg%3E)
Hanafi Ons
0
Reputation points
commented 2025-07-02T06:59:08.9633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 50%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Venkat Reddy Navari
3,060
Reputation points Microsoft External Staff
Moderator
1 answer
No response to HDInsight quota request – SR# 2506200030006435 (Southeast Asia)
Hi Microsoft Team, I had submitted a quota increase request to enable 10 cores for HDInsight cluster in Southeast Asia under Pay-As-You-Go subscription. S: 2506200030006435 . Raised on: June 20, 2025 . Subscription: ******@gmail.com I received a…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-06-24T06:14:16.2233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 43%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVM%3C/text%3E%3C/svg%3E)
Vinaykumar Marali
20
Reputation points
answered 2025-07-01T04:32:56.45+00:00
Vinaykumar Marali
20
Reputation points
1 answer
One of the answers was accepted by the question author.
Quota approved but still unable to create HDInsight cluster — no available cores
Hi Team, I raised a request for HDInsight core quota increase and got confirmation via email that the quota was approved for AV2 and Dads v5 families. However, I am still unable to create even the minimum configuration HDInsight cluster. I tried…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-06-25T07:15:21.57+00:00
Vinaykumar Marali
20
Reputation points
accepted 2025-06-27T10:57:25.7433333+00:00
Vinaykumar Marali
20
Reputation points
1 answer
NullPointerException in RegionSizeCalculator when using Hive Warehouse Connector on HDInsight (5.1) / Azure Spark
Hello, I'm encountering a NullPointerException when running a Spark job on Azure HDInsight (Spark with Hive Warehouse Connector). The job fails during input split generation from an HBase table. The relevant portion of the log is: 25/06/01 06:28:18 ERROR…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-06-23T13:43:11.2466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 96%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EET%3C/text%3E%3C/svg%3E)
ECSEKI Tibor
5
Reputation points
commented 2025-06-25T19:00:30.41+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
14,585
Reputation points Microsoft External Staff
Moderator
1 answer
Issue with On Demand HDInsight in Hive Activity.
I have been facing this for a lot of days now, I am trying to use the Hive Activity in ADF to create and On Demand HDInsight Cluster, with no customization, no VNET or subnet. But for some reason I am still facing the issue of the Provisioning Failing…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-06-13T06:47:28.4366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 46%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
Vishesh Agarwal
0
Reputation points
edited an answer 2025-06-17T05:16:31.6233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 80%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Krupal Bandari
770
Reputation points Microsoft External Staff
Moderator
1 answer
HD Insights não me disponibiliza nós para começar um cluster
Não consigo criar nenhum nó no HDInsights. Nenhuma região possui nós disponíveis. Preciso de cotas
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-06-03T21:08:50.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 64%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGB%3C/text%3E%3C/svg%3E)
Gustavo Bonifácio Pereira Guida
0
Reputation points
commented 2025-06-05T17:09:33.4633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
7,295
Reputation points Microsoft External Staff
Moderator
1 answer
HDInsight cluster creation Fails
{ "code": "InvalidNetworkConfigurationErrorCode", "message": "Virtual Network configuration is incompatible with HDInsight requirements. Error: 'Cluster Validation timed out.', Please follow…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-05-01T18:57:29.3733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 96%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
Kavin Kingslin
0
Reputation points
answered 2025-05-02T07:29:16.65+00:00
Krupal Bandari
770
Reputation points Microsoft External Staff
Moderator
2 answers
Unable to delete HDInsight cluster and resource group.
Unable to delete HDInsight cluster and resource group. I am getting and error and the deletion process keeps running indefinitely.
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-04-17T20:44:02.2033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 24%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Abhinandan Gupta
0
Reputation points
answered 2025-05-01T11:15:49.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 57.00000000000001%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDR%3C/text%3E%3C/svg%3E)
Dileep Raj Narayan Thumula
330
Reputation points Microsoft External Staff
Moderator
1 answer
I want to delete my HDInsight cluster
I cannot ssh into the HD cluster, and Ambari UI is not loading either. The cluster has the status deallocated. I want to delete the cluster and create a new one
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-03-25T21:29:32.3133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 5%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIM%3C/text%3E%3C/svg%3E)
Isaac McSey
0
Reputation points
commented 2025-03-27T19:07:47.3033333+00:00
Chandra Boorla
14,585
Reputation points Microsoft External Staff
Moderator
1 answer
How to Disable or Remove Hive Warehouse Connector from HDInsight 5.1
Hello, We are running Spark on HDInsight 5.1 and are not using Hive Metastore at all. However, our Spark application is failing with the following exception: User class threw exception: java.lang.NoClassDefFoundError:…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-03-12T11:51:57.01+00:00
ECSEKI Tibor
5
Reputation points
commented 2025-03-14T07:33:53.6866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
24,260
Reputation points Microsoft External Staff
Moderator
1 answer
Azure HDInsight Cluster Not getting Deleted
Hi !!! I have a HDInsight Cluster which I am not able to delete. It is running for more than 30 mins. The name of the cluster is "sayakban-cc-spark-cluster". This is a high priority task as it is consuming my credits from my educational…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-03-03T23:24:43.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 97%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Sayak Banerjee
0
Reputation points
edited an answer 2025-03-12T06:47:17.5+00:00
Ganesh Gurram
7,295
Reputation points Microsoft External Staff
Moderator
0 answers
HDInsight Spark-type cluster failing to be deployed due to failed start up of SPARK3_THRIFTSERVER service
In Microsoft Azure the Spark-type HDInsight cluster (with 5.1 stack version) is failing to be deployed with an AmbariClusterCreationFailedErrorCode error-code due to an internal error of Spark3 hriftserver service on Ambari: Execute['!…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-02-24T15:42:41.6233333+00:00
Tibor, Ecseki
0
Reputation points
commented 2025-02-27T12:38:37.71+00:00
Chandra Boorla
14,585
Reputation points Microsoft External Staff
Moderator
1 answer
Creating Azure HDInsight process is stuck at "Status: HDInsight Configuration"
Hey, During the creation of Azure HDinsight (Hadoop Cluster) The process seems to be stuck at "Status: HDInsight Configuration" For more than 20 minutes; after it did manage to somewhat create it (note the dashboard itself is up and running),…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-02-13T09:40:47.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 42%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
lidorbenshitrit
0
Reputation points
commented 2025-02-17T11:53:28.9766667+00:00
Ganesh Gurram
7,295
Reputation points Microsoft External Staff
Moderator
2 answers
Issue Creating HDInsight Cluster with Virtual Network
I am trying to create an HDInsight cluster using Terraform. However, I encounter the following error when connecting it to a virtual network: │ Error: creating Hadoop HDInsight Cluster (Subscription: "XXX" │ Resource Group Name:…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2025-01-28T18:41:16.3233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 13%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEJ%3C/text%3E%3C/svg%3E)
Emily Jil Sauer
0
Reputation points
commented 2025-02-17T09:40:38.7333333+00:00
AnnuKumari-MSFT
34,556
Reputation points Microsoft Employee
Moderator
1 answer
Sku family standardev3family quota is not enough
I'm attempting to go through this tutorial: https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-jupyter-spark-sql-use-portal?source=docs I cannot create a HDInsight cluster because of this error that I'm seeing on the Configuration +…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2024-12-19T18:33:00.0833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 9%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Matt
0
Reputation points
commented 2024-12-23T02:43:17.32+00:00
Ganesh Gurram
7,295
Reputation points Microsoft External Staff
Moderator
1 answer
The resource api version is unavailable, how to delete resource
the service corresponding to the resource located under sub has been taken offline or the API version is unavailable, which makes it impossible to delete this resource. How to delete the reources in portal? The VM network and other resources…
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 80%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EXW%3C/text%3E%3C/svg%3E)
1 answer
In Data Lake Storage Account Gen 2 unexpected reduction of calls API "GetPathStatus" and "ListFilesystemDir"
Hi, I am reaching out with an inquiry regarding the transactions quantity in an data lake storage account gen 2 on Azure. We have noticed that starting on 28th october, the transactions quantity associated with storage account have significantly…
Azure Storage
Azure Storage
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,535 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2024-11-12T11:14:53.7933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 81%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Javier Coll Doñetz
0
Reputation points
commented 2024-11-25T05:56:16.9266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
8,181
Reputation points Microsoft External Staff
Moderator
0 answers
Unable to create HDInsight On Demand Cluster in Azure Data Factory
I am getting the issue to creating the On Demand HDInsights in Azure Data Factory. The status is stuck at "PreparingCluster". Here's my LinkedService configuration. OS type : Linux
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
asked 2024-11-13T03:52:15.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 19%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETC%3C/text%3E%3C/svg%3E)
Tan Check Leam
10
Reputation points
commented 2024-11-22T01:25:18.93+00:00
Chandra Boorla
14,585
Reputation points Microsoft External Staff
Moderator
1 answer
One of the answers was accepted by the question author.
Compatibility of Azure HDInsight Kafka
Can you provide clarity on whether Azure HDInsight Kafka works with Azure Event Registry? Additionally, I’m interested in any compatibility details or constraints that might apply. Could you share more on this integration?
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,633 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 32%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
Difference between synapse and hdinsight
What are major difference between synapse and HD insight and under what circumstances one should use synapse and vice versa
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
231 questions
asked 2021-10-12T12:43:25.457+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 34%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Shivam Ramola
71
Reputation points
commented 2024-11-06T12:09:12.1433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 92%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Michael Schmidt
365
Reputation points