HBase Accelerated Writes in HDInsight, Failure Recovery

Hi @prathap sagar

Thank you for contacting to Microsoft QA, below are the few detailed steps to mitigate the reported steps -

You are exploring how to enable HBase Accelerated Writes in Azure HDInsight and have questions about durability, performance trade-offs, and recovery. Let’s walk through each point:

What happens if the local disk copy fails before syncing to ADLS Gen2?

If the local Write Ahead Log (WAL) gets corrupted or fails before syncing to ADLS Gen2, that data could be lost—unless it’s replicated. HDInsight helps reduce this risk by keeping three copies of the WAL on different managed disks. So, unless all three copies fail at the same time, the chance of losing data is very low.

Is there a possibility of data loss?

Yes, but it’s rare. Data loss would only occur if all three WAL copies fail simultaneously, which is unlikely because HDInsight automatically replicates WAL data across multiple disks.

Does HDInsight rely only on local WAL?

No. It adds redundancy by replicating the WAL across three disks connected to different VMs, which improves durability.

How does recovery work?

If there’s a failure, HBase uses the WAL to replay recent updates that weren’t flushed to disk. Thanks to replication, the system can recover using these WAL copies.

How does write-latency improvement compare to durability risk?

Accelerated Writes can make writes 4–10 times faster by using premium SSD-managed disks instead of writing directly to cloud storage. This is great for performance, but it does introduce some durability risk if all WAL copies fail. It’s best suited for write-heavy workloads where speed is critical.

What kind of performance gains should you expect?

Expect a big boost—writes can be 4–10 times faster, which is especially helpful for applications that handle a lot of writes.

Is Accelerated Writes recommended for all workloads?

Not necessarily. It’s most useful for write-intensive workloads. If your app is mostly read-heavy, you probably don’t need it.

Additional Tips:

• Make sure your cluster has at least three worker nodes for durability and recovery.
• Always flush and disable HBase tables before making changes to avoid losing WAL data.

References:

• Azure HDInsight Accelerated Writes for Apache HBase
• Troubleshoot Apache HBase
• Migrate Apache HBase to a new version
• Set up backup and replication for Apache HBase

Enabling HBase Accelerated Writes in HDInsight offers significant performance improvements by writing the Write Ahead Log (WAL) to premium SSD-managed disks instead of cloud storage. However, there are important considerations regarding durability and recovery:

1. What happens if the local disk copy fails or becomes corrupted before the asynchronous sync to ADLS Gen2 occurs?
   • Possibility of Data Loss: If the local disk copy fails or becomes corrupted before the asynchronous sync to Azure Data Lake Storage (ADLS) Gen2 occurs, there is a risk of data loss. The WAL serves as a commit log, and if updates are only present in the local WAL and not yet synced, they could be lost if the disk fails.
   • Redundancy Maintenance: HDInsight does not rely solely on the local WAL for durability. The Accelerated Writes feature is designed to improve write performance, but it is crucial to ensure that data is eventually synced to a more durable storage solution like ADLS Gen2 to maintain redundancy.
   • Recovery Process: In the event of a failure, recovery would typically involve replaying the WAL to restore the data to the last committed state, but this is contingent on the WAL being intact and not corrupted.
2. How does the write-latency improvement compare against the potential durability or recovery risk?
   • Performance Gains: Users can expect significant improvements in write-latency due to the use of premium SSD-managed disks, which provide excellent I/O performance. This is particularly beneficial for write-intensive workloads, as it helps mitigate the bottleneck caused by writing to cloud storage.
   • Workload Recommendations: Accelerated Writes are generally recommended for workloads that are write-intensive and can tolerate some risk of data loss in the event of a local disk failure. However, for critical data where durability is paramount, it is advisable to ensure that data is regularly synced to ADLS Gen2 or implement additional redundancy measures.

In summary, while Accelerated Writes can enhance performance, careful consideration should be given to the potential risks of data loss and the importance of syncing to durable storage solutions for critical applications.

References:

• Azure HDInsight Accelerated Writes for Apache HBase
• Troubleshoot Apache HBase performance issues on Azure HDInsight