SharePoint 2019 Crawling Issue

Hi Shane Rogers

Based on the details you've shared, the key to this entire problem is the error message you received for site1: This item was deleted because it was excluded by a crawl rule". Since your other sites are crawling correctly, this is an issue isolated to the configuration related to site1 (on port 39064).

Check Crawl Rules (The Primary Suspect)

This directly addresses the error message. A misconfigured rule is the most common reason for this behavior.

• Navigate to Central Administration > Manage service applications > Click on the name of your new Search Service Application.
• On the left, under Crawling, click on Crawl Rules.
• Carefully examine the list for any Exclusion rules. Look for a broad rule like http://servername:* or one that could unintentionally match http://servername:39064*.

Please note that SharePoint applies the first matching rule it finds, so the order is critical.

Here are recommendations you can consider:

• Option A: Create a new, more specific Include rule for your site (ex: http://servername:39064/*. Make sure this new rule is ordered before any broad exclusion rule.
• Option B: Edit or delete the problematic exclusion rule if you are certain, it is not needed.

After making a change, you must run a Full Crawl on your content source to see the results. Check for a Preventive Robots Directive (robots.txt)

The generic part of the error message mentions this. It appears to be the second strongest possibility.

• On your SharePoint server, open Internet Information Services (IIS) Manager.
• Expand Sites, find the IIS site running on port 39064, right-click it, and select Explore.
• Look for a file named robots.txt in the root folder. If it exists, open it and check for a line like Disallow: /, which blocks all crawlers.

Here’s how you can address this:

• You can rename the file (ex: robots.txt.old) or delete it.
• If you need the file for other purposes, you can add a specific entry to allow the SharePoint crawler:

Verify Site and Web Application Settings

Site-Level Searchability:

• Browse to http://servername:39064.
• Go to Site Settings > Search and offline availability.
• Ensure "Allow this site to appear in search results" is set to Yes.

Alternate Access Mappings (AAM):

• In Central Administration, go to Application Management > Configure alternate access mappings.
• Select the Web Application for site1 and ensure http://servername:39064 is listed in the Default zone. The crawler requires this for proper access.

If you are still having issues, this diagnostic step will prove whether the problem is rule-based. You can try:

• In your Search Service Application, create a new, temporary Content Source that contains only the URL for site1.
• Temporarily disable or delete all Crawl Rules.
• Run a Full Crawl on this new content source only.

If the crawl succeeds, one of your previous configurations (either a crawl rule or robots.txt) was the problem.

Additionally, I found another user experiencing behavior similar to yours. You may also refer to their recommended solution, which involves manually uninstalling all missing features:

stsadm -o uninstallfeature -id <GUID> -force

Once that step was completed, the crawler resumed its activity.

For your referral: search - The item was deleted because it was excluded by a crawl rule - SharePoint Stack Exchange

(Note: This information is provided as a convenience to you. These sites are not controlled by Microsoft, and Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. Please ensure that you fully understand the risks before using any suggestions from the above link).

Please understand that we're not Microsoft support, this is a user-to-user support forum. Moderators have no backend access and cannot directly intervene in Microsoft products. We provide only technical guidance and best-practice recommendations based on reported issues.

I hope this information helps and if you have any further questions, please feel free to ask via comment section!

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.