Share via


MSExchangeFastSearch Warning 1009 - But ContentIndexState Healthy

Question

Wednesday, September 27, 2017 8:41 AM

Hi,

Running three Exchange Server 2016 Version 15.1 (Build 1034.26). All in one DAG.

EXCH1, EXCH2, EXCH3 - All Databases were active on EXCH1 at the time.

I receive following two eventlog entries +/- once a day at 3am from EXCH3: 

03:07:44

The indexing of mailbox database STAFF encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. > Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. > Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: An exception occurred processing MDB: 612a4ccd-492d-4cc4-88fc-b990d26acc89. > Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: Failed to read notifications, MDB: 612a4ccd-492d-4cc4-88fc-b990d26acc89. > Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)
Diagnostic context:
    Lid: 33865   StoreEc: 0x476     
    Lid: 1494     Remote Context Beg 
    Lid: 41980   dwParam: 0x3
    Lid: 38140   StoreEc: 0x476     
    Lid: 34221   StoreEc: 0x476     
    Lid: 1750     Remote Context End 
    Lid: 38985   StoreEc: 0x476     
    Lid: 20098  
    Lid: 20585   StoreEc: 0x476     
   at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)
   at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)
   at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.<>c__DisplayClass3.<ReadEvents>b__1()
   at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass4`1.<TranslateMapiExceptionsWithRetryAndReturnValue>b__3()
   at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass7.<TranslateMapiExceptionsWithRetry>b__6()
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
    End of inner exception stack trace 
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithRetry(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall, Action reconnectAction, MdbHealthTracker mdbHealthTracker)
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithRetryAndReturnValue[TReturnValue](IDiagnosticsSession tracer, LocalizedString errorString, Func`1 mapiCall, Action reconnectAction, MdbHealthTracker mdbHealthTracker)
   at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadEvents(Int64 startCounter, Int32 eventCountWanted, ReadEventsFlags flags, Int64& endCounter)
   at Microsoft.Exchange.Search.Mdb.NotificationsFeeder.PollForEvents()
   at Microsoft.Exchange.Search.Mdb.NotificationsFeeder.<>c__DisplayClass6.<ProcessingProcedure>b__5()
   at Microsoft.Exchange.Search.Core.Common.Executable.TryRunUnderExceptionHandler(Action action, LocalizedString message)
    End of inner exception stack trace 
    End of inner exception stack trace 
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SearchFeedingController.ExecuteComplete(IAsyncResult asyncResult)
    End of inner exception stack trace 
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SyncRootController.ExecuteComplete(IAsyncResult asyncResult)


03:13:14
The indexing of mailbox database STAFF encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. > Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: Failed to read notifications, MDB: 612a4ccd-492d-4cc4-88fc-b990d26acc89. > Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)
Diagnostic context:
    Lid: 33865   StoreEc: 0x476     
    Lid: 1494     Remote Context Beg 
    Lid: 41980   dwParam: 0x3
    Lid: 38140   StoreEc: 0x476     
    Lid: 34221   StoreEc: 0x476     
    Lid: 1750     Remote Context End 
    Lid: 38985   StoreEc: 0x476     
    Lid: 20098  
    Lid: 20585   StoreEc: 0x476     
   at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)
   at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)
   at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.<>c__DisplayClass3.<ReadEvents>b__1()
   at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass4`1.<TranslateMapiExceptionsWithRetryAndReturnValue>b__3()
   at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass7.<TranslateMapiExceptionsWithRetry>b__6()
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
    End of inner exception stack trace 
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithRetry(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall, Action reconnectAction, MdbHealthTracker mdbHealthTracker)
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithRetryAndReturnValue[TReturnValue](IDiagnosticsSession tracer, LocalizedString errorString, Func`1 mapiCall, Action reconnectAction, MdbHealthTracker mdbHealthTracker)
   at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadEvents(Int64 startCounter, Int32 eventCountWanted, ReadEventsFlags flags, Int64& endCounter)
   at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadFirstEventCounter()
   at Microsoft.Exchange.Search.Engine.NotificationsEventSourceInfo..ctor(IWatermarkStorage watermarkStorage, INotificationsEventSource eventSource, IDiagnosticsSession diagnosticsSession, MdbInfo mdbInfo)
   at Microsoft.Exchange.Search.Engine.SearchFeedingController.DetermineFeederStateAndStartFeeders()
   at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart()
   at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state)
    End of inner exception stack trace 
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SyncRootController.ExecuteComplete(IAsyncResult asyncResult)

And following eventlog entry +/- once a day at 3am from EXCH2:

03:07:27

The indexing of mailbox database TEACHER encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. > Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. > Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: An exception occurred processing MDB: 3b0f02ab-6b17-4a5a-839c-a52c0744b8f1. > Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: Failed to read notifications, MDB: 3b0f02ab-6b17-4a5a-839c-a52c0744b8f1. > Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)
Diagnostic context:
    Lid: 33865   StoreEc: 0x476     
    Lid: 1494     Remote Context Beg 
    Lid: 41980   dwParam: 0x3
    Lid: 38140   StoreEc: 0x476     
    Lid: 34221   StoreEc: 0x476     
    Lid: 1750     Remote Context End 
    Lid: 38985   StoreEc: 0x476     
    Lid: 20098  
    Lid: 20585   StoreEc: 0x476     
   at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)
   at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)
   at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.<>c__DisplayClass3.<ReadEvents>b__1()
   at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass4`1.<TranslateMapiExceptionsWithRetryAndReturnValue>b__3()
   at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass7.<TranslateMapiExceptionsWithRetry>b__6()
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
    End of inner exception stack trace 
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithRetry(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall, Action reconnectAction, MdbHealthTracker mdbHealthTracker)
   at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithRetryAndReturnValue[TReturnValue](IDiagnosticsSession tracer, LocalizedString errorString, Func`1 mapiCall, Action reconnectAction, MdbHealthTracker mdbHealthTracker)
   at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadEvents(Int64 startCounter, Int32 eventCountWanted, ReadEventsFlags flags, Int64& endCounter)
   at Microsoft.Exchange.Search.Mdb.NotificationsFeeder.PollForEvents()
   at Microsoft.Exchange.Search.Mdb.NotificationsFeeder.<>c__DisplayClass6.<ProcessingProcedure>b__5()
   at Microsoft.Exchange.Search.Core.Common.Executable.TryRunUnderExceptionHandler(Action action, LocalizedString message)
    End of inner exception stack trace 
    End of inner exception stack trace 
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SearchFeedingController.ExecuteComplete(IAsyncResult asyncResult)
    End of inner exception stack trace 
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SyncRootController.ExecuteComplete(IAsyncResult asyncResult)

I now (6 hours later) checked  "Get-MailboxDatabaseCopyStatus | ft name,contentindexstate" - Healthy on all DBs and all Servers.

What does the Warning mean? Is it possible it recovered by itself?

Regards, Eve Meier

All replies (2)

Wednesday, September 27, 2017 11:15 AM

Hi

Kindly stop Microsoft Exchange search as well Exchange search host controller. Check the blog

https://practical365.com/exchange-server/fix-failed-database-content-index-exchange-2013/ to fix it.

Regards,
Raj
________________________________________________________________________________
Please mark as an "Answer" and do "Vote",if my contribution or answer is useful


Thursday, September 28, 2017 11:06 AM

Hi,

Based on my search, I find an issue with the same error message, and this issue occurs because of corrupted Content Index Catalog.

To solve the issue, we could use one of the two methods:

  1. Reseed the Search Catalog by running:
Update-MailboxDatabaseCopy -Identity <DB name>\<server name>  -CatalogOnly
  1. Manual rebuild the Content Index: 
    1> Stop the “Microsoft Exchange Search” service and the “Microsoft Exchange Host Controller” service.
    2> Delete the Existing CI catalog folder.
         Go to C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox database Name, find the Existing CI catalog folder and delete it. See as follow:

    3> Start the “Microsoft Exchange Search service” and the “Microsoft Exchange Host Controller” service.

If issue remains, run the following command and let me know the result.

Get-DatabaseAvailabilityGroup | select -ExpandProperty:Servers | Test-ReplicationHealth

Best Regards,

Manu Meng
TechNet Community Support

Please remember to mark the replies as answers if they help and unmark them if they provide no help.
If you have feedback for TechNet Subscriber Support, contact [email protected].