Azure Event Hubs with Kafka, read_committed consumers stuck

Hi **MALINGE Benjamin,
**I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

Hello,

Thank you very much for your detailed explanation and guidance.

We are currently in the process of opening a more in-depth support ticket for the engineering team in the Azure Subscription.

---

New Test :

After performing additional tests, it appears that Azure Event Hubs correctly handles automatic transaction abortion after the configured timeout when the message is successfully sent. In these cases, the abort marker is properly emitted and the LSO advances as expected.

First Test

// First send 
kafkaProducer.initTransactions(); 
kafkaProducer.beginTransaction(); 
kafkaProducer.send(producerRecord); 
// No commit !!

// Second send with different producer
kafkaProducer2.initTransactions(); 
kafkaProducer2.beginTransaction(); 
kafkaProducer2.send(producerRecord2); 
kafkaProducer2.commitTransaction();

In this example, the consumer eventually consumes the second send after Azure Event Hubs automatically rolls back the first transaction (after the configured transaction timeout).

Second Test

However, we performed a second test that highlights a different behavior between Strimzi and Azure Event Hubs in a specific scenario.

Test scenario

Using a Spring Kafka KafkaTemplate with @Transactional:

• Send message 1
• Wait (sleep) for a duration greater than transaction.timeout.ms
• Send message 2 (using the same producer)

Result with Strimzi

• First send: OK
• After transaction.timeout.ms, the Strimzi broker automatically writes an abort marker for the transaction
• Second send: FAILS with: org.apache.kafka.common.errors.InvalidProducerEpochException: Producer attempted to produce with an old epoch.
• Attempt to abort the transaction from the producer: FAILS with: org.apache.kafka.common.errors.ProducerFencedException: There is a newer producer with the same transactionalId which fences the current one.

Final state on the broker is consistent:

• The first message is properly aborted
• No zombie transaction or message
• Subsequent messages are consumed normally

Result with Azure Event Hubs

• First send: OK
• After transaction.timeout.ms, the Event Hubs Kafka broker automatically writes an abort marker for the transaction
• Second send: SUCCEEDS (unexpected) → Is this behavior normal?
• Attempt to commit the transaction from the producer: FAILS with: org.apache.kafka.common.errors.ProducerFencedException: There is a newer producer with the same transactionalId which fences the current one.

Final state on the broker is inconsistent:

• The first message is correctly aborted
• However, the second message is accepted and never receives an abort marker, either from the broker or the producer
• This results in a message without any subsequent transactional marker

As a consequence:

• Consumers using read_committed are blocked
• The LSO (Last Stable Offset) does not progress beyond this message

Analysis

The observed behavior on Event Hubs suggests that:

• Transaction expiration is correctly handled by the broker
• However, subsequent Produce requests are not systematically rejected (unlike a native Kafka implementation such as Strimzi)
• Validation of the transactional state appears to occur late, at the EndTxn (commit/abort) stage rather than at Produce time

---

Original issue

The original issue we observed appears in a more specific and less controlled scenario compared to the second test described above.

It occurs when message sends fail consecutively due to timeouts (with 3 retries, according to our configuration), and the producer eventually exceeds the default delivery.timeout.ms (120s).

In this situation:

• The producer attempts to abort the transaction
• However, the abort operation fails because the producer has been fenced (a new producer instance is created by Spring Kafka with the same transactional.id)

Unlike in our second test, in this original scenario, we initially assumed that no message had been successfully sent before the transaction failure occurred.

Additional hypothesis

However, based on our recent findings, we are now questioning this assumption.

It is possible that:

• Some send attempts may actually reach the broker and start the transaction, even if they eventually timeout on the client side
• In other words, a send can:
  • fail from the client perspective (timeout)
  • but still have partially succeeded on the broker side

If this is the case, then:

• The last send attempt (after retries) could behave similarly to the second send in our controlled test
• That is, it may be accepted by the broker even though the transaction has already expired or is in an invalid state

Observed impact

From that point onward, we observe that:

• Consumers remain stuck in read_committed
• The behavior suggests the presence of a “ghost” or incomplete transaction
• The Last Stable Offset (LSO) does not progress

This would be consistent with the behavior observed in the second test:

• a message being accepted after transaction expiration
• without receiving a proper abort marker
• effectively blocking consumers

Current limitation

The main challenge is that we have not yet been able to consistently reproduce this exact original scenario on demand, which makes deeper investigation more difficult.

---

Question

Could you please confirm whether:

• the behavior observed in Azure Event Hubs — specifically the fact that a send can still succeed after the transaction has expired (as demonstrated in our second test), whereas it is rejected in a standard Kafka implementation such as Strimzi — is expected?
• or if this represents a known limitation or divergence from standard Kafka transactional semantics?

---

Thank you again for your help and guidance.

Best regards, Benjamin

Hi Benjamin,

Thank you for the detailed follow-up and for sharing both test scenarios — your analysis is thorough and very helpful.

Based on your observations, the behavior you’re seeing is due to a difference between native Apache Kafka (e.g., Strimzi) and Azure Event Hubs Kafka endpoint in how transactional state is enforced.

In native Kafka:

The broker validates the producer epoch and transaction state during the Produce request

If a transaction has expired, subsequent sends are rejected immediately

In Azure Event Hubs (Kafka API):

Transaction validation is primarily enforced at the EndTxn stage (commit/abort)

As a result, a Produce request may still succeed even after the transaction has expired

If the producer is later fenced, the commit/abort fails and no transaction marker is written

This can lead to:

Records written without a corresponding commit/abort marker

Last Stable Offset (LSO) not advancing

• read_committed consumers appearing stuck, even though higher offsets exist

Yes — this is a known behavioral difference (limitation) in Azure Event Hubs Kafka compatibility.

While Event Hubs supports Kafka transactions, it does not fully replicate all broker-side enforcement semantics of Apache Kafka, particularly around Produce-time validation after transaction expiry.

To avoid this scenario and ensure stable consumer behavior:

• Keep transactions short-lived
  Avoid long delays (such as sleep) within an active transaction
• Align timeout settings carefully
  Ensure:
  • delivery.timeout.ms is less than or equal to transaction.timeout.ms
  • Retry windows do not extend beyond transaction validity
• Handle fencing errors as terminal
  For exceptions like:
  • ProducerFencedException
  • InvalidProducerEpochException
    → Immediately close and recreate the producer, and avoid further sends in that transaction

Limit producer retries
Reducing retries helps prevent late sends being accepted after the transaction has already expired
From your test results, the scenario where a Produce request succeeds after transaction expiry but does not result in a valid transaction marker is consistent with current Azure Event Hubs Kafka behavior.

This can lead to “incomplete” transactional records and LSO blocking for read_committed consumers, as you observed. Aligning timeout settings, minimizing retries, and strictly handling producer fencing will help prevent this condition and ensure consumers continue processing as expected.

Microsoft Reference Links:

• Azure Event Hubs – Kafka Transactions
  https://learn.microsoft.com/azure/event-hubs/apache-kafka-transactions
• Azure Event Hubs – Kafka Overview (feature behavior and differences)
  https://learn.microsoft.com/azure/event-hubs/apache-kafka-overview
• Apache Kafka Transactions (reference for native Kafka behavior)
  https://kafka.apache.org/documentation/#transactions

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

Please do not forget to "Accept Answer" and "up-vote" wherever the information provided helps you, as this can be beneficial to other community members.