SharePoint 2013 Crawl Rules not Excluding Search Results but matching the rules

This issue happened with me in SharePoint 2013 however i believe the resolution would be the same for SharePoint 2010 and SharePoint 2016 environments.

Problem:

I have SharePoint 2013 environment and i need to remove all the search results under a specific URL. I created a Crawl Rule to Exclude content, in few cases it did not excluded any content and sometimes it excluded only a partial content. I ran an Incremental Crawl each time i create any rule but same results.

Scenario: 

I have site collection under https://webapplication/pa/b and i want to remove all the search results under this URL. This is how i created the rule.

Created the above rule and press OK.

Now for testing if this rules matches our query, i have provided it a sample URL and it matches one of our rule. However, when i ran incremental crawl, the target URL was still appearing in my search results which i didn't want.

Solution:

If it matches your rules but do not exclude from search results, it means its not being crawled in your Incremental Crawl. It is understood that SharePoint do not crawl each item in Incremental Crawls so you have to manually set that URL to be crawled in next Incremental Crawl.

Go to Search Service Application > Crawl Log > URL View  and put the target URL in it and press Search button. It will show all the results under that URL. Now click the root URL in the results, click the item and click "Recrawl this item in the next crawl".  See below screenshot for more information. 

 Now run the Incremental Crawl on your content source.

After the crawl completes successfully, it should not show any search results under that URL. If you still see any URL in search results, make sure your no other rule is including it or you can copy that exact URL and put it in URL View to check the status of that specific item.

Happy SharePointing :)