Connecting to an HDInsight Kafka Cluster from a Local Machine via Python Script

Question

Connecting to an HDInsight Kafka Cluster from a Local Machine via Python Script

Hanafi Ons 0

Hello,

I've configured my Kafka cluster on Azure HDInsight with a virtual network (VNet) that includes a NAT gateway and a Network Security Group (NSG). Now, I’m trying to connect to this cluster from my local Windows machine using a Python script to send messages.

I am unsure how to correctly configure the bootstrap.servers parameter in my script. Could you provide guidance or examples on the appropriate format or settings for this scenario?

Thank you!

1 answer

Your answer

Answer 1

Venkat Reddy Navari 3,060 Microsoft External Staff Moderator

Hi Hanafi Ons Since your HDInsight Kafka cluster is deployed inside a VNet with a NAT Gateway and NSG, connecting directly from your local machine isn't straightforward the Kafka brokers have internal IPs not accessible externally by default.

Option 1: SSH Tunnel via Edge Node or VM (Recommended for Dev/Test)

Deploy an edge node or use a VM in the same VNet as the Kafka cluster.
SSH into that node and forward a local port to a broker port using:
```
   
   ssh -i path/to/your/key.pem -L 9093:<broker-hostname>:9093 sshuser@<edge-node-public-ip>
```
This tunnels your local port 9093 to the Kafka broker inside the VNet.
In your Python script, set bootstrap.servers to localhost:9093:


Copy
from kafka import KafkaProducer

producer = KafkaProducer(
    bootstrap_servers='localhost:9093',
    security_protocol='SSL',  # if SSL is enabled on HDInsight Kafka
    ssl_cafile='ca-cert.pem',
    ssl_certfile='client-cert.pem',
    ssl_keyfile='client-key.pem'
)

Option 2: Public Access via Load Balancer (Use with Caution)

If security policies allow, expose the Kafka brokers via a public IP or Azure Load Balancer:

Update NSG rules to allow inbound access to Kafka ports (e.g., 9093).
Set the Kafka brokers’ advertised.listeners to use the public IP.
Then use:


bootstrap_servers='<public-ip>:9093'

This is not recommended for production due to security implications unless secured properly with SSL and authentication.

Option 3: Connect via VPN or ExpressRoute

If your local environment is connected to the same VNet via a site-to-site VPN or ExpressRoute, you can access the private IPs of the Kafka brokers directly.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Hanafi Ons 0 Reputation points

2025-07-02T00:24:12.5866667+00:00

Thank you, Venkat, for the detailed explanation and options. I’ve been trying to configure a Point-to-Site (P2S) VPN to connect to the HDInsight Kafka cluster, but unfortunately, I couldn't even ping the worker nodes (kafka brokers).

Based on your guidance, it seems that my P2S VPN setup might not be the right approach, and using the other options could be more effective. Could you please confirm if P2S doesn't work in this purpose?

Thanks again!
Venkat Reddy Navari 3,060 Reputation points Microsoft External Staff Moderator

2025-07-02T06:59:08.9633333+00:00
Hi Hanafi Ons
You're very welcome happy to help.

You're right to suspect the P2S VPN may not be working as expected in this case. By default, Point-to-Site VPN can work for accessing HDInsight Kafka brokers, but only if all of the following are correctly configured:

Checklist for P2S VPN to Access Kafka Brokers:

VNet Gateway: Ensure the VNet where HDInsight is deployed has a configured VPN Gateway (not just a NAT Gateway).

Address Space: Your P2S client IP range must not conflict with the VNet’s IP range.

Routing:

The VPN client must have proper routes to reach the HDInsight subnet.

You can check this using route print (on Windows) after connecting.

NSG Rules: The NSGs on the HDInsight subnet or NICs must allow inbound traffic from your P2S client IP range on Kafka ports (e.g., 9092, 9093).

Brokers' Advertised Hostnames: Kafka brokers typically advertise internal DNS names make sure your machine can resolve those names. You may need to:

Use a custom DNS server in your VNet that your client also uses, or

Add broker hostnames and IPs to your local hosts file (not ideal for dynamic IPs).

Common Pitfall: Even if the VPN connects, if DNS resolution or NSG rules block access, you won’t be able to ping or connect to broker ports, which is likely what you’re facing now.

Recommendation:

If fixing the P2S setup seems too complex or unreliable for now, I'd recommend going with:

Option 1: SSH Tunnel via edge node in the same VNet easiest path for secure dev/test.

Option 3: ExpressRoute or site-to-site VPN for production-grade connectivity.

Hope this helps. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Connecting to an HDInsight Kafka Cluster from a Local Machine via Python Script

1 answer

Your answer