MongoDB Container persisted to Azure Blob (Container) Storage get corrupted

Question

MongoDB Container persisted to Azure Blob (Container) Storage get corrupted

Arnoldas Karpavicius 5

Hi,

I am using “mongo:noble” container for my AKS project, the container is persisted to Azure Container Storage ( Blob storage ). It was working fine till last month, but then suddenly when the AKS was loading the MongoDB container it started failing and complaining that the database is corrupted.

PersistentVolume class used: “azureblob-fuse-premium” csi driver: blob.csi.azure.com storage capacity: 100Gi ( actually used like 10gb )

attaching the log from the MongoDB

{"t":{"$date":"2025-04-07T12:55:34.218+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}

{"t":{"$date":"2025-04-07T12:55:34.221+00:00"},"s":"I", "c":"CONTROL", "id":5945603, "ctx":"main","msg":"Multi threading initialized"}

{"t":{"$date":"2025-04-07T12:55:34.221+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set at least one of the related parameters","attr":{"relatedParameters":["tcpFastOpenServer","tcpFastOpenClient","tcpFastOpenQueueSize"]}}

{"t":{"$date":"2025-04-07T12:55:34.223+00:00"},"s":"I", "c":"NETWORK", "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":25},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":25},"outgoing":{"minWireVersion":6,"maxWireVersion":25},"isInternalClient":true}}}

{"t":{"$date":"2025-04-07T12:55:34.223+00:00"},"s":"I", "c":"TENANT_M", "id":7091600, "ctx":"main","msg":"Starting TenantMigrationAccessBlockerRegistry"}

{"t":{"$date":"2025-04-07T12:55:34.223+00:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/mongoaz","architecture":"64-bit","host":"db-8677c79db9-pm67q"}}

{"t":{"$date":"2025-04-07T12:55:34.223+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"8.0.6","gitVersion":"80f21521ad4a3dfd5613f5d649d7058c6d46277f","openSSLVersion":"OpenSSL 3.0.13 30 Jan 2024","modules":[],"allocator":"tcmalloc-google","environment":{"distmod":"ubuntu2404","distarch":"x86_64","target_arch":"x86_64"}}}}

{"t":{"$date":"2025-04-07T12:55:34.223+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"24.04"}}}

{"t":{"$date":"2025-04-07T12:55:34.223+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"storage":{"dbPath":"/data/mongoaz"}}}}

{"t":{"$date":"2025-04-07T12:55:34.303+00:00"},"s":"I", "c":"STORAGE", "id":22270, "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/mongoaz","storageEngine":"wiredTiger"}}

{"t":{"$date":"2025-04-07T12:55:35.911+00:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"55EBF17C36E8","b":"55EBEB9BC000","o":"5E076E8","s":"wiredtiger_open","s+":"26F8"}}}

`{"t":{"$date":"2025-04-07T12:55:35.911+00:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"55EBF1758201","b":"55EBEB9BC000","o":"5D9C201","s":"ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8","C":"mongo::WiredTigerKVEngine::_openWiredTiger(std::__cxx11::basic_string

Hari Babu Vattepally 2,560 Reputation points Microsoft External Staff

2025-04-10T11:52:43.7666667+00:00

Hi @Arnoldas Karpavicius ,

Thanks for sharing details with logs, could you please confirm whether this issue is primarily affecting MongoDB, or if other services using the same setup are also experiencing problems? Identifying this will help narrow down the root cause.
Alex Burlachenko 4,225 Reputation points

2025-04-10T12:06:40.97+00:00
Hi Arnoldas,

Thank you for reaching out and sharing your question on the Q&A portal. I appreciate the detailed logs and context you provided—it’s very helpful for troubleshooting.

From the logs, it looks like the MongoDB container is encountering a segmentation fault (Signal: 11) when trying to read data from the Azure Blob Storage-backed persistent volume. This could indeed be related to configuration or compatibility issues between MongoDB’s WiredTiger storage engine and the Azure Blob FUSE driver.

A few things to check:

Ensure the azureblob-fuse-premium storage class is configured with the recommended settings for MongoDB workloads. Blob FUSE may not always handle frequent small I/O operations as efficiently as MongoDB expects.

Verify the MongoDB version (8.0.6) is fully compatible with the underlying OS and storage driver. Sometimes, edge cases arise with specific combinations.

Check if the corruption persists when using a different storage class (e.g., Azure Disk) temporarily to isolate the issue.

Let me know if you’d like to dive deeper into any of these areas. Happy to help further!

Best regards,

Alex

P.S. If my answer helps you, please Accept my answer.
Arnoldas Karpavicius 5 Reputation points

2025-04-10T12:52:11.4466667+00:00

Hi Alex,

Thanks for your reply! Unfortunately, I could not find much information about persisting MongoDB to Azure Blob Storage. As such I kept the configuration more or less as in the Microsoft documentation on how to attach a blob-fuse PVC.

The same storage is also attached to a few other deployments ( same AKS ), an API that uses it for file-storage and a Tomcat that loads .war files from and has H2 database on. It seems that those 2 containers are not encountering any issues.

Originally, this MongoDB container was using Azure Files for persistence and it worked fine there, but as Tomcat container was not able to load .war files from there I moved it all to Container storage ( all related to same project, so wanted to keep data in same storage ).

Attaching PV/PVC configurations:

apiVersion: v1

kind: PersistentVolume

metadata:

name: focus-pv

spec:

capacity:

storage: 100Gi

accessModes:

- ReadWriteMany

persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion

storageClassName: azureblob-fuse-premium

mountOptions:

- -o allow_other

- --file-cache-timeout-in-seconds=120

csi:

driver: blob.csi.azure.com

# volumeHandle has to be unique for every identical storage blob container in the cluster

# character `#`and `/` are reserved for internal use and cannot be used in volumehandle

volumeHandle: "storagetest01_demostorage"

volumeAttributes:

containerName: "demostorage"

nodeStageSecretRef:

name: storage-secret

namespace: default

---

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: datavol

spec:

accessModes:

- ReadWriteMany

volumeName: focus-pv

storageClassName: azureblob-fuse-premium

resources:

requests:

storage: 100Gi

---

Initially I had the storage at 4Gi, but after it initially failed and I repaired the DB I increased it to 100Gi, actual size in the storage is around 5Gb

Attaching statistics of the storage container in case it helps:

Finished calculating statistics for 'storagetest01/demostorage/'. Completed 10/04/2025 13:45. Active blobs: 132 blobs, 140.83 MiB (147,676,156 bytes). Snapshots: 0 blobs, 0 B (0 bytes). Versions: 92,898 blobs, 57.65 GiB (61,904,813,616 bytes). Deleted blobs: 0 blobs, 0 B (0 bytes). Total: 93,030 items, 57.79 GiB (62,052,489,772 bytes).

In regards to Blob-fuse not always handling frequent small I/Os, would NFS version help with that?

Kind regards,

Arnoldas
Arnoldas Karpavicius 5 Reputation points

2025-04-10T12:57:43.7333333+00:00

Hi @Hari Babu Vattepally ,

As mentioned in a comment to @Alex Burlachenko , there are multiple containers sharing the PVC, and from all of them only MongoDB seems to fail. Also I have a separate copy of the project with same setup and that one had failed too.
Arnoldas Karpavicius 5 Reputation points

2025-04-10T13:00:59.17+00:00

Hi,

I cant seem to Edit the post so add a little bit of additional information. The corruption of the DB is not exactly consistent. Sometimes it fails after a restart, sometimes it stays alive for a few days. The DB in question worked after restart and the next day ( we only keep the AKS resource online during working hours ), but failed after the weekend.

Kind regards,

Arnoldas
Alex Burlachenko 4,225 Reputation points

2025-04-10T15:45:49.5466667+00:00
Hi Arnoldas,

Yes, using Azure Blob NFS (Network File System) instead of Blob FUSE would likely resolve the frequent small I/O issues you're seeing with MongoDB. NFS Is Better for MongoDB Than Blob FUSE. u know why?

NFS (v3.0) is a true file system protocol, optimized for low-latency, random read/write operations exactly what MongoDB’s WiredTiger engine needs.

Blob FUSE, while flexible, is still an emulation layer over object storage and struggles with high-frequency small I/O.

NFS avoids the metadata overhead and caching quirks of Blob FUSE, reducing corruption risks.

It supports file locking and strong consistency, critical for MongoDB’s journaling and checkpointing.

Many users run MongoDB successfully on NFS-backed storage (including Azure NFS), whereas Blob FUSE is more suited for large, infrequently accessed files (like your Tomcat .war files).

Actaully you can test it

Create an NFS-enabled Blob Container but with premium tier only (sorry if non)

az storage account create \ --name <storage-account-name> \ --resource-group <resource-group> \ --location <region> \ --sku Premium_LRS \ --enable-nfs-v3 true

for full setup manual https://learn.microsoft.com/en-us/azure/storage/blobs/network-file-system-protocol-support

Update Your PV/PVC to use NFS instead of Blob FUSE

apiVersion: v1 kind: PersistentVolume metadata: name: mongodb-nfs-pv spec: capacity: storage: 100Gi accessModes: - ReadWriteMany storageClassName: "" mountOptions: - nfsvers=3 nfs: server: <storage-account-name>.blob.core.windows.net path: "/<container-name>"

No need for CSI drivers, NFS mounts natively in AKS.

Test MongoDB’s stability, deploy with the NFS volume and monitor for crashes or corruption. If NFS Blob isn’t available in your region, consider Azure Files with NFS offers similar performance benefits.

https://learn.microsoft.com/en-us/azure/storage/files/storage-files-quick-create-use-linux

So... Blob FUSE is great for Tomcat/API (large, static files).

Blob NFS/Azure Files NFS is ideal for MongoDB (high-frequency small I/O).

Best regards,

Alex

P.S. If my answer help to you, please Accept my answer
Vijayalaxmi Kattimani 2,225 Reputation points Microsoft External Staff

2025-04-11T12:03:38.8133333+00:00

Hi Arnoldas Karpavicius,

Following up to see, we haven’t heard from you on the last response provided by Alex Burlachenko,

In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Vijayalaxmi Kattimani 2,225 Reputation points Microsoft External Staff

2025-04-14T08:58:07.1266667+00:00

Hi @Arnoldas Karpavicius,

We haven’t heard from you on the last response and wanted to follow up to check if your issue has been resolved.

If you have found a solution, we would appreciate it if you could share it with the community, as it may be helpful to others. Otherwise, please provide more details, and we will do our best to assist you further.
Arnoldas Karpavicius 5 Reputation points

2025-04-14T13:01:00.59+00:00

Hi @Vijayalaxmi Kattimani ,

In the process of testing it.

Notes so far: The Bash script to create NFSv3 storage didn't work. Received an error that "IsNFSv3Storage" cannot be true.

Currently trying to work out permissions seems right now containers accessing the data are throwing "PERMISSION DENIED" error. As soon as I am able to properly test it, I will update/comment the post!

Thanks!

Kind regards,

Arnoldas

Your answer

Hari Babu Vattepally 2,560 Reputation points Microsoft External Staff

2025-04-10T11:52:43.7666667+00:00

Hi @Arnoldas Karpavicius ,

Thanks for sharing details with logs, could you please confirm whether this issue is primarily affecting MongoDB, or if other services using the same setup are also experiencing problems? Identifying this will help narrow down the root cause.
Alex Burlachenko 4,225 Reputation points

2025-04-10T12:06:40.97+00:00

Hi Arnoldas,

Thank you for reaching out and sharing your question on the Q&A portal. I appreciate the detailed logs and context you provided—it’s very helpful for troubleshooting.

From the logs, it looks like the MongoDB container is encountering a segmentation fault (Signal: 11) when trying to read data from the Azure Blob Storage-backed persistent volume. This could indeed be related to configuration or compatibility issues between MongoDB’s WiredTiger storage engine and the Azure Blob FUSE driver.

A few things to check:

Ensure the azureblob-fuse-premium storage class is configured with the recommended settings for MongoDB workloads. Blob FUSE may not always handle frequent small I/O operations as efficiently as MongoDB expects.

Verify the MongoDB version (8.0.6) is fully compatible with the underlying OS and storage driver. Sometimes, edge cases arise with specific combinations.

Check if the corruption persists when using a different storage class (e.g., Azure Disk) temporarily to isolate the issue.

Let me know if you’d like to dive deeper into any of these areas. Happy to help further!

Best regards,

Alex

P.S. If my answer helps you, please Accept my answer.
Arnoldas Karpavicius 5 Reputation points

2025-04-10T12:57:43.7333333+00:00

Hi @Hari Babu Vattepally ,

As mentioned in a comment to @Alex Burlachenko , there are multiple containers sharing the PVC, and from all of them only MongoDB seems to fail. Also I have a separate copy of the project with same setup and that one had failed too.
Arnoldas Karpavicius 5 Reputation points

2025-04-10T13:00:59.17+00:00

Hi,

I cant seem to Edit the post so add a little bit of additional information. The corruption of the DB is not exactly consistent. Sometimes it fails after a restart, sometimes it stays alive for a few days. The DB in question worked after restart and the next day ( we only keep the AKS resource online during working hours ), but failed after the weekend.

Kind regards,

Arnoldas
Vijayalaxmi Kattimani 2,225 Reputation points Microsoft External Staff

2025-04-11T12:03:38.8133333+00:00

Hi Arnoldas Karpavicius,

Following up to see, we haven’t heard from you on the last response provided by Alex Burlachenko,

In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Vijayalaxmi Kattimani 2,225 Reputation points Microsoft External Staff

2025-04-14T08:58:07.1266667+00:00

Hi @Arnoldas Karpavicius,

We haven’t heard from you on the last response and wanted to follow up to check if your issue has been resolved.

If you have found a solution, we would appreciate it if you could share it with the community, as it may be helpful to others. Otherwise, please provide more details, and we will do our best to assist you further.
Arnoldas Karpavicius 5 Reputation points

2025-04-14T13:01:00.59+00:00

Hi @Vijayalaxmi Kattimani ,

In the process of testing it.

Notes so far: The Bash script to create NFSv3 storage didn't work. Received an error that "IsNFSv3Storage" cannot be true.

Currently trying to work out permissions seems right now containers accessing the data are throwing "PERMISSION DENIED" error. As soon as I am able to properly test it, I will update/comment the post!

Thanks!

Kind regards,

Arnoldas

Share via

MongoDB Container persisted to Azure Blob (Container) Storage get corrupted

Your answer