Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Question
Thursday, October 18, 2018 12:40 PM
Hi,
Having an issue with FastSearch on Exchange 2016 CU10. Clients are unable to search in Outlook and get the error "Something went wrong and your search couldn't be completed."
Server side I'm getting a lot of different errors. When restarting the 'MSExchangeFastSearch' & 'HostControllerService' services
An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. > System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.tcp://localhost:3803/Management/IndexController that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.
Server stack trace:
at System.ServiceModel.Channels.ConnectionUpgradeHelper.DecodeFramingFault(ClientFramingDecoder decoder, IConnection connection, Uri via, String contentType, TimeoutHelper& timeoutHelper)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Ceres.SearchCore.Admin.IIndexManagement.get_IndexSystemNames()
at Microsoft.Exchange.Search.Fast.IndexManager.<>c.<GetCatalogs>b__43_0(IIndexManagement agent)
at Microsoft.Exchange.Search.Fast.FastManagementClient.PerformFastOperation[TManagementAgent,TResult](AgentHandle`1 agentHandle, Func`2 function, String eventLogKey)
End of inner exception stack trace
swiftly followed by
The indexing of mailbox database 'dbname' encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. > Microsoft.Exchange.Search.Engine.FeedingSkippedException: "Feeding was skipped for 'a073537b-979a-438a-8fdf-6c29256d4450 ('dbname')' due to the state 'Unknown', error code: 'Unknown'."
at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart()
at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state)
End of inner exception stack trace
at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Engine.SyncRootController.ExecuteComplete(IAsyncResult asyncResult)
and finally
The indexing of mailbox database 'dbname' encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. > Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. > Microsoft.Exchange.Search.Fast.FastTransientDocumentException: The Content Submission Service returned failure for the document: The Content Submission Service returned failure for the document: Item operation failed after 0 retries
.. > Microsoft.Exchange.Search.Fast.FastTransientDocumentException: The Content Submission Service returned failure for the document: Item operation failed after 0 retries
.
End of inner exception stack trace
at Microsoft.Exchange.Search.Fast.FastFeeder.EndSubmitDocument(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Mdb.NotificationsFeeder.DocumentCompleteCallback(IAsyncResult asyncResult)
End of inner exception stack trace
at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Engine.SearchFeedingController.ExecuteComplete(IAsyncResult asyncResult)
End of inner exception stack trace
at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
I've tried:
- Stopping both search services, renaming the catalog, starting the services again
- Renaming the node folder in "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
Then uninstalling and reinstalling the search config:
.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
- Reseeding the catalog.
- Manually creating the ContentSubmitters AD group and granting full control to Administrators and Network Service.
Strangely search in OWA does work - so I'm presuming it uses another mechanism.
Any help would be appreciated.
All replies (5)
Friday, October 19, 2018 7:58 AM
Hi,
What is your Outlook build number? Ensure that your Outlook has all the patches.
What is your database content index status currently? Run the following command and post the output here:
Get-MailboxDatabaseCopyStatus | fl content*
Regards,
Manu Meng
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.
Friday, October 19, 2018 8:30 AM
Hi there,
Testing in Outlook version 1807 Build 10325.20118.
Exchange side DB's look like this:
ContentIndexState : Crawling
ContentIndexErrorMessage : The Microsoft Exchange Search Service is crawling the database.
ContentIndexErrorCode : 3
ContentIndexVersion : 22
ContentIndexCrawlVersion : 0
ContentIndexBacklog :
ContentIndexRetryQueueSize :
ContentIndexMailboxesToCrawl : 80
ContentIndexSeedingPercent :
ContentIndexSeedingSource :
ContentIndexServerSource :
ContentIndexState : Failed
ContentIndexErrorMessage : An internal error occurred for the database or its index.
ContentIndexErrorCode : 2
ContentIndexVersion :
ContentIndexCrawlVersion :
ContentIndexBacklog :
ContentIndexRetryQueueSize :
ContentIndexMailboxesToCrawl :
ContentIndexSeedingPercent :
ContentIndexSeedingSource :
ContentIndexServerSource :
ContentIndexState : Failed
ContentIndexErrorMessage : An internal error occurred for the database or its index.
ContentIndexErrorCode : 2
ContentIndexVersion :
ContentIndexCrawlVersion :
ContentIndexBacklog :
ContentIndexRetryQueueSize :
ContentIndexMailboxesToCrawl :
ContentIndexSeedingPercent :
ContentIndexSeedingSource :
ContentIndexServerSource :
It's probably worth noting that the index state of the databases is fluctuating from Crawling to Failed quite a lot.
In the search diagnostic logs locate here: C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Diagnostics\Logs
I'm seeing a lot of errors like this:
10/19/2018 09:08:35.66 Microsoft.Exchange.Search.Servi (0x186C) 0x8BD8 Search Exchange a9k6e Monitorable Watermark document: [IndexSystem:60F445C7-A59D-42C0-A5FA-44E39EDBE14D12, IndexId:0, Watermark: 48878653] failed with exception: Microsoft.Exchange.Search.Fast.FastFeedingAffinityMismatchException: The Content Submission Service returned failure for the document: The Content Submission Service returned failure for the document: The item could not be indexed successfully because the item failed in the indexing subsystem. Content group fK0SpW10.45 was submitted in generation GID[6974]. Strict feeding affinity mismatch between operation affinity:ed6f15c6-83c6-4a06-bfad-763d829e3dc9 and index router id:4cbe660a-c793-473a-9228-17ebf3ec15cd. .. > Microsoft.Exchange.Search.Fast.FastFeedingAffinityMismatchException: The Content Submission Service returned failure for the document: The item co...
10/19/2018 09:08:35.66* Microsoft.Exchange.Search.Servi (0x186C) 0x8BD8 Search Exchange a9k6e Monitorable ...uld not be indexed successfully because the item failed in the indexing subsystem. Content group fK0SpW10.45 was submitted in generation GID[6974]. Strict feeding affinity mismatch between operation affinity:ed6f15c6-83c6-4a06-bfad-763d829e3dc9 and index router id:4cbe660a-c793-473a-9228-17ebf3ec15cd. . End of inner exception stack trace at Microsoft.Exchange.Search.Fast.FastFeeder.EndSubmitDocument(IAsyncResult asyncResult) at Microsoft.Exchange.Search.Fast.WatermarkFeeder.WatermarkFeederAsyncResult.Callback(IAsyncResult ar)]
Wednesday, October 24, 2018 2:53 AM
Hi there,
Testing in Outlook version 1807 Build 10325.20118.
Exchange side DB's look like this:
ContentIndexState : Crawling ContentIndexErrorMessage : The Microsoft Exchange Search Service is crawling the database. ContentIndexErrorCode : 3 ContentIndexVersion : 22 ContentIndexCrawlVersion : 0 ContentIndexBacklog : ContentIndexRetryQueueSize : ContentIndexMailboxesToCrawl : 80 ContentIndexSeedingPercent : ContentIndexSeedingSource : ContentIndexServerSource : ContentIndexState : Failed ContentIndexErrorMessage : An internal error occurred for the database or its index. ContentIndexErrorCode : 2 ContentIndexVersion : ContentIndexCrawlVersion : ContentIndexBacklog : ContentIndexRetryQueueSize : ContentIndexMailboxesToCrawl : ContentIndexSeedingPercent : ContentIndexSeedingSource : ContentIndexServerSource : ContentIndexState : Failed ContentIndexErrorMessage : An internal error occurred for the database or its index. ContentIndexErrorCode : 2 ContentIndexVersion : ContentIndexCrawlVersion : ContentIndexBacklog : ContentIndexRetryQueueSize : ContentIndexMailboxesToCrawl : ContentIndexSeedingPercent : ContentIndexSeedingSource : ContentIndexServerSource :It's probably worth noting that the index state of the databases is fluctuating from Crawling to Failed quite a lot.
In the search diagnostic logs locate here: C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Diagnostics\Logs
I'm seeing a lot of errors like this:
10/19/2018 09:08:35.66 Microsoft.Exchange.Search.Servi (0x186C) 0x8BD8 Search Exchange a9k6e Monitorable Watermark document: [IndexSystem:60F445C7-A59D-42C0-A5FA-44E39EDBE14D12, IndexId:0, Watermark: 48878653] failed with exception: Microsoft.Exchange.Search.Fast.FastFeedingAffinityMismatchException: The Content Submission Service returned failure for the document: The Content Submission Service returned failure for the document: The item could not be indexed successfully because the item failed in the indexing subsystem. Content group fK0SpW10.45 was submitted in generation GID[6974]. Strict feeding affinity mismatch between operation affinity:ed6f15c6-83c6-4a06-bfad-763d829e3dc9 and index router id:4cbe660a-c793-473a-9228-17ebf3ec15cd. .. > Microsoft.Exchange.Search.Fast.FastFeedingAffinityMismatchException: The Content Submission Service returned failure for the document: The item co... 10/19/2018 09:08:35.66* Microsoft.Exchange.Search.Servi (0x186C) 0x8BD8 Search Exchange a9k6e Monitorable ...uld not be indexed successfully because the item failed in the indexing subsystem. Content group fK0SpW10.45 was submitted in generation GID[6974]. Strict feeding affinity mismatch between operation affinity:ed6f15c6-83c6-4a06-bfad-763d829e3dc9 and index router id:4cbe660a-c793-473a-9228-17ebf3ec15cd. . End of inner exception stack trace at Microsoft.Exchange.Search.Fast.FastFeeder.EndSubmitDocument(IAsyncResult asyncResult) at Microsoft.Exchange.Search.Fast.WatermarkFeeder.WatermarkFeederAsyncResult.Callback(IAsyncResult ar)]
Is there a DAG?
Regards,
Manu Meng
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.
Monday, October 29, 2018 2:33 PM
Databases are in a DAG.
I managed to get the content indexing status to Healthy again by editing the configuration files located %ExchangeInstallPath%\Bin\Search\Ceres\HostController\Data\Nodes\Fsis\NODENAME\Configuration\Local
Inside WcfConfigurator.xml I removed the <AuthorizedRole>ContentSubmitters</AuthorizedRole> tag.
Whilst this had got the indexing status back to healthy, search still fails in Outlook 365.
Wednesday, October 31, 2018 8:00 AM
Databases are in a DAG.
I managed to get the content indexing status to Healthy again by editing the configuration files located %ExchangeInstallPath%\Bin\Search\Ceres\HostController\Data\Nodes\Fsis\NODENAME\Configuration\Local
Inside WcfConfigurator.xml I removed the <AuthorizedRole>ContentSubmitters</AuthorizedRole> tag.
Whilst this had got the indexing status back to healthy, search still fails in Outlook 365.
Is it a hybrid environment? How about using Outlook 2016?
Regards,
Manu Meng
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].
Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.