Exchange 2019 CU14: Server-side search is very slow or fails after mailbox migration from Exchange 2013

Hello,

We are troubleshooting a server-side search performance issue after migrating mailboxes from Exchange 2013 to Exchange 2019.

Environment

• Exchange 2013 CU23 still exists in coexistence.
• Exchange 2019 CU14, build 15.2.1544.36.
• Two Exchange 2019 DAG members: <Exchange Server 1> and <Exchange Server 2>.
• Mailbox databases are replicated between both Exchange 2019 servers.
• MAPI/HTTP is disabled.
• The problem reproduces in OWA, so it is not limited to Outlook, OST files, or Windows Search.

Main affected mailbox

• Mailbox size: approximately 74 GB
• Approximately 236,000 items
• Current database: <Mailbox Database>

The mailbox was originally located on Exchange 2013.

It was migrated to Exchange 2019, then moved back to Exchange 2013 for troubleshooting, and later migrated again to Exchange 2019.

This provided a very clear A/B test:

• Exchange 2013: Search is fast and works correctly.
• Exchange 2019: Search is consistently much slower. Some searches eventually return results, while others hang for several minutes and fail.
• Moved back to Exchange 2013: Search becomes fast again.
• Moved again to Exchange 2019: Slow/failing search behavior returns.

So the behavior follows the Exchange version rather than the Outlook client.

Search symptoms

Search performance on Exchange 2019 is generally slow for the affected larger mailboxes.

Some queries eventually complete, while others remain running for a long time and eventually fail.

Examples:

<Reference ID 1>              -> very slow / eventually fails
<Reference Number 1>                 -> very slow / eventually fails
subject:"<Reference Number 1>"       -> very slow / eventually fails
<Order ID 1>             -> very slow / eventually fails
CRM:<Record ID 1>           -> very slow / eventually fails
<Reference ID 2>                  -> very slow / eventually fails

<Reference ID 3>    -> completes, but search is still slow
<Reference ID 4>                 -> completes, but search is still slow
<Reference ID 5>                   -> completes, but search is still slow

The search scope also affects the behavior.

For example:

<Reference ID 1> + Inbox only
-> can eventually return results after a long delay

<Reference ID 1> + All folders
-> usually hangs and eventually fails

The same behavior is reproducible in OWA.

After long-running searches, OWA can display:

The server is busy and can't respond to your request.
Please try again later.

X-OWA-Error:
Microsoft.Exchange.Data.Directory.SystemConfiguration.OverBudgetException

X-OWA-Version: 15.2.1544.36

We currently believe that OverBudgetException is probably secondary rather than the original problem.

The search operation can remain active for tens of seconds or several minutes. Repeated or long-running search requests eventually consume the OWA workload budget, after which OWA returns OverBudgetException.

Microsoft Troubleshoot-ModernSearch results

We ran the Microsoft CSS Troubleshoot-ModernSearch script against one of the failing queries:

.\Troubleshoot-ModernSearch.release.ps1 `
  -MailboxIdentity "<affected mailbox>" `
  -ItemSubject "<Reference ID 1>" `
  -MatchSubjectSubstring `
  -QueryString "<Reference ID 1>"

The script found 9 different messages with the subject RA-18877.

The target messages report:

IndexStatus            : Indexed
IsPermanentFailure     : NULL
IsPartiallyIndexed     : False
IndexingErrorMessage   : NULL
BigFunnelPOIIsUpToDate : True

Mailbox BigFunnel statistics are approximately:

AssociatedItemCount               : 108
ItemCount                         : 236664
BigFunnelMessageCount             : 236400
BigFunnelIndexedCount             : 224405
BigFunnelPartiallyIndexedCount    : 11984
BigFunnelNotIndexedCount          : 0
BigFunnelCorruptedCount           : 0
BigFunnelStaleCount               : 11

For every RA-18877 target item, the diagnostic query returns:

BigFunnelMatchPOI    = True
BigFunnelMatchFilter = BigFunnelMatchFilter Failed

together with:

Get-StoreQuery DiagnosticQueryException:
ErrorCode: NotFound
LID: 64548
Filter doesn't exits

We do not know whether:

LID 64548 - Filter doesn't exits

is related to the actual search problem or is only an artifact of the diagnostic BigFunnelMatchFilter operation.

The fact that:

BigFunnelMatchPOI = True

for all target items appears to confirm that the search terms exist in the indexed POI.

Troubleshooting already performed

1. Mailbox move / recreation of BigFunnel state

The mailbox was moved from Exchange 2013 to Exchange 2019 again.

The migration completed successfully and a new BigFunnel mailbox state was created on the Exchange 2019 database, but the search problem remained.

Therefore, simply moving the mailbox and recreating its BigFunnel state did not resolve the issue.

2. DAG database switchover

The active database copy was switched between both Exchange 2019 DAG members.

The same mailbox/database was therefore tested with Store/Search running on both Exchange 2019 servers.

The behavior changed slightly between servers, but the underlying issue remained.

On one server, Inbox-only searches were somewhat more likely to complete, but mailbox-wide searches still remained very slow or failed.

3. Search services restarted

We restarted:

MSExchangeFastSearch
HostControllerService

No meaningful improvement.

4. BigFunnelPOI mailbox repair

We ran:

New-MailboxRepairRequest `
  -Mailbox "<affected mailbox>" `
  -CorruptionType BigFunnelPOI

Result:

JobState            : Succeeded
Progress            : 100
CorruptionsDetected : 0
CorruptionsFixed    : 0
ErrorCode           :

Search behavior did not change.

5. Search Folder workaround

We temporarily tested the following Variant Configuration override:

Component:
BigFunnel

Section:
BigFunnelDiscoveryQuerySettings

Parameter:
NoSearchFolder=false

The override was successfully applied and refreshed.

There was no meaningful improvement, so the override was removed.

6. Third-party module warning

Troubleshoot-ModernSearch reported:

Search Process Status:
Warning - Third Party Modules Detected

Investigation showed that the only non-Microsoft module loaded in:

Microsoft.Exchange.Search.Service

was:

Google.Protobuf.ni.dll
Company: Google Inc.
FileVersion: 3.7.0.0

The same module is present on both Exchange 2019 servers, so this does not appear to be AV/EDR injection.

7. ExchangeLogCollector

We collected:

• Search logs
• Search diagnostic logs
• BigFunnel/Search logs
• OWA logs
• IIS logs

using Microsoft CSS ExchangeLogCollector.

During failing searches, OWA ExecuteSearch requests spend approximately 30-60 seconds or sometimes several minutes in the Exchange backend.

There is no corresponding IIS 503 or mailbox database failure.

Some failed requests eventually result in an OWA application-level error rather than an HTTP transport failure.

Cancelling a search does not always appear to immediately terminate the already running backend operation.

Mailbox size observation

We have also tested other migrated mailboxes.

Current observation:

~27 GB mailbox
-> search completes without an error, although Exchange 2019 search is still slower than Exchange 2013

~45 GB mailbox
-> search is slow and can fail

~74 GB mailbox
-> search is very slow and frequently fails

We do not conclude that mailbox size itself is necessarily the root cause.

It may instead correlate with:

• item count;
• number of candidate documents;
• posting list size;
• mailbox-wide query cost;
• BigFunnel query execution;
• another scalability issue in Exchange 2019 Search.

However, the difference between Exchange 2013 and Exchange 2019 is very clear.

The same large mailbox searches quickly while hosted on Exchange 2013 and becomes slow again after being moved to Exchange 2019.

Questions

1. Is there a known Exchange 2019 BigFunnel/Search issue where larger mailboxes migrated from Exchange 2013 experience very slow server-side search?
2. What exactly does the following diagnostic error mean?

      Get-StoreQuery DiagnosticQueryException
      ErrorCode: NotFound
      LID: xxx
      Filter doesn't exits
   

3. Is BigFunnelMatchFilter Failed significant when the same item reports:

      BigFunnelMatchPOI = True
      IndexStatus = Indexed
      BigFunnelPOIIsUpToDate = True
   

4. If the target items are fully indexed and BigFunnelPOI repair reports zero corruption, what is the recommended next diagnostic step for a query that spends several minutes inside ExecuteSearch?
5. Is there a supported method to completely rebuild the search/BigFunnel state of an individual Exchange 2019 mailbox beyond a mailbox move and BigFunnelPOI repair?
6. Are there known Exchange 2019 query scalability issues where larger mailboxes cause server-side search to run long enough to eventually exhaust the OWA workload budget?
7. Could this behavior be related to a known Exchange 2019 CU14 Search issue that was corrected in a later build?

We can provide the complete output from:

• Troubleshoot-ModernSearch
• ExchangeLogCollector

including Search, BigFunnel, OWA and IIS diagnostic logs if required.

Thank you.