Azure SQL Change Event Streaming: producer stops delivering to one Event Hubs partition after a transient outage and never reconnects

Question

Azure SQL Change Event Streaming: producer stops delivering to one Event Hubs partition after a transient outage and never reconnects

Eugenio Rossetto 0

Environment

Azure SQL Database, General Purpose (reproduced on both serverless and provisioned compute), CES preview enabled.

Destination: Event Hubs Standard namespace, Kafka endpoint (mynamespace.servicebus.windows.net:9093/myhub), @destination_type = N'AzureEventHubs', Microsoft Entra managed-identity credential, @partition_key_scheme = N'Table', JSON encoding, 4 partitions, 14 tables in one stream group.

Scenario

After a transient connectivity interruption, delivery to one partition fails while the other partitions keep delivering normally. If the interruption is short (~8 minutes in one captured episode), the producer recovers on its own. If it lasts longer (~35 minutes in a contrasting episode with the identical error signature), the producer never recovers: it retries the same batch every 5 seconds against the dead connection indefinitely (29 hours in our worst case), batch_start_lsn in sys.dm_change_feed_log_scan_sessions and the checkpoint (seqno) stay frozen, and the healthy partitions receive the same window re-published as duplicates (~1,300 msg/h). The only recovery is sp_disable_change_event_stream plus a full group rebuild. We have documented 14 episodes of this over four weeks.

Errors (repeating every ~5 s in sys.dm_change_feed_errors; two variants)

23692 Delivery of the message failed due to a timeout...
23679 Unable to reach the configured destination.
23658 Change Event Streaming encountered a SQL exception.
22782 Failed to output batch data for table N in partition XXXX
22783 Failed to process commit work item
22777 Failure reported in table group / database processing

Troubleshooting done

Event Hubs healthy throughout every episode: zero ServerErrors/UserErrors/ThrottledRequests, namespace Active. We enabled namespace diagnostic logs (KafkaCoordinatorLogs, KafkaUserErrorLogs, OperationalLogs, DiagnosticErrorLogs) and captured an episode live: while CES reported error 23679, Event Hubs logged zero errors and no connection activity from the producer. Other connections from the same producer deliver to the same namespace during the failure, and rebuilding the group with identical configuration and credential restores delivery instantly, so network, firewall and authentication are ruled out. Reproduced on serverless (0.5 and 1 vCore minimum) and provisioned (GP_Gen5_2) compute. No Resource Health or activity-log events in any window. No CDC, replication, Synapse Link or mirroring on the database. Followed the configure guide and its limitations section. Failures concentrate on one partition per period (same partition id across consecutive incidents); onsets show no time-of-day pattern and historically cluster around the first writes after idle stretches.

Question

Is this a known defect in the CES preview producer's connection recovery, and is there a supported way to make the producer re-establish a failed partition connection without sp_disable_change_event_stream (which destroys the group, the table enrollment and the error ring buffer, and skips the wedged partition's undelivered events past the checkpoint reset)?

We have full captures (error ring buffers, scan sessions, checkpoints, Event Hubs diagnostic logs) for several episodes and can share them.

Vinodh247-1375 44,801 Reputation points Volunteer Moderator

2026-09-23T00:55:18.23+00:00
Are Azure SQL db and Eventhubs in the same region, and what networking configuration is used, such as public access, private endpoints, selected networks or firewall rules?

With @partition_key_scheme = N'Table', which tables map to the affected partition, and does the failure consistently follow a specific table or partitionID?

Across the 14 incidents, is there a consistent idle duration or outage threshold after which automatic recovery no longer occurs?

During a permanently wedged episode, did any non-destructive operation, such as scaling, pause/resume or database failover, cause CES to establish a new kafka connection, or is disabling and rebuilding the stream the only action that works?
Eugenio Rossetto 0 Reputation points

2026-09-23T03:04:17.8666667+00:00
@Vinodh247-1375 Thanks for the follow-up. Answers below.

1. Region and networking. Azure SQL Database and the Event Hubs namespace are both in France Central (same region). Networking is public on both sides: SQL publicNetworkAccess = Enabled, restrictOutboundNetworkAccess = Disabled, no private endpoints; the only relevant firewall rule is ALLOW_AZURE (0.0.0.0). Event Hubs namespace is Standard tier, publicNetworkAccess = Enabled, no private endpoint, no IP filtering. So CES reaches the hub over the public Kafka endpoint (:9093), which is the only supported path.

2. Table to partition mapping, it follows the partition, not a table. With @partition_key_scheme = N'Table', the failure is consistently pinned to one partition id per stream-group generation, and that id is stable for the whole life of the group but changes when the stream is rebuilt:

-1864006014, 5 consecutive incidents, Sep 5 to 11

1016301186, 2 incidents, Sep 13 to 14 (on provisioned compute)

2023261826, 2 incidents, Sep 21 to 22

Within a generation, the 22782 "table N in partition X" messages name whichever enrolled tables were written during that incident and hash to the bad partition. The table ordinals shift between incidents (0,1,2,3,4 seen), but the partition id is invariant until a rebuild. So the failure follows the partition, not any specific table. This strongly suggests the stuck connection is the producer's connection to one specific partition/broker endpoint, not anything table-related.

3. Idle/outage threshold, we cannot cleanly characterize it, and here is why. Across 16 episodes we have never observed the producer self-recover from a fully frozen checkpoint. Every wedge required a stream rebuild (manual at first, automated on a 10-minute watchdog since Sep 4). Episodes do pass through a degraded phase first (errors on the failing partition every ~5s while the checkpoint still crawls on the others), and some of those degraded phases clear while others progress to a full freeze, but we have not instrumented the exact degraded-duration cutoff that predicts which way it goes. We can add that instrumentation and report the threshold if it would help. What we can say firmly: once the checkpoint is fully frozen, it never recovered on its own in any of the 16 cases.

4. Non-destructive operations, untested, and we would value your guidance here. We have only ever recovered by disabling and rebuilding the stream (sp_disable_change_event_stream + recreate group + re-enroll tables). We have not tried scaling, pause/resume, or a database failover during a wedged episode, so we cannot say whether any of those forces the producer to establish a fresh Kafka connection. Given the partition-pinned pattern in (2), a failover (which would move the primary and necessarily re-establish connections) seems like the most informative test. If you can confirm that a failover is expected to reset the CES producer connections, we will try it on the next wedge instead of rebuilding, and report whether it clears without the data gap that a rebuild causes.

We have full captures (error ring buffers, scan sessions, checkpoints, and Event Hubs KafkaCoordinator/Operational diagnostic logs showing zero producer activity during a wedge) and can share them through a support case if useful.

1 answer

Your answer

Vinodh247-1375 44,801 Reputation points Volunteer Moderator

2026-09-23T00:55:18.23+00:00

Are Azure SQL db and Eventhubs in the same region, and what networking configuration is used, such as public access, private endpoints, selected networks or firewall rules?

With @partition_key_scheme = N'Table', which tables map to the affected partition, and does the failure consistently follow a specific table or partitionID?

Across the 14 incidents, is there a consistent idle duration or outage threshold after which automatic recovery no longer occurs?

During a permanently wedged episode, did any non-destructive operation, such as scaling, pause/resume or database failover, cause CES to establish a new kafka connection, or is disabling and rebuilding the stream the only action that works?

Answer 1

Mladen Andzic - Msft 11 Microsoft Employee

Hi @Eugenio Rossetto , greetings from the SQL product group. I'd suggest opening a support case so the feature team can investigate it. I'm sure they'll get to the bottom of it.

0 comments