Try our conversational search powered by Generative AI!

Ben Nitti
Oct 6, 2021
  2684
(1 votes)

How to exclude pages from your search index, sitemaps and internet search engines

Your client wants to exclude certain pages from showiing up in your site's search results, they also want prevent these pages from being crawled by external search engines (Google, Bing etc.) and lastly they want them removed from their sitemap.xml file.  All of this can be achieved and controlled with a checkbox property on the Settings tab in the editor. I'll demonstrate below. 

Create an interface with a boolean property that inherits IContent

	public interface IDisableIndexing : IContent
	{
		bool DisableIndex { get; set; }
	}

Add it to your base class or any page type 

    public abstract class SitePageData : PageData, IDisableIndex
    {
        [CultureSpecific]
		[Display(Name = "Disable Indexing",
			Description = "Removes the page from search index, sitemap and search engines",
			GroupName = SystemTabNames.Settings,
			Order = 10)]
		public virtual bool DisableIndex { get; set; }
    }

Optimizely Search & Navigation

You can filter  from your Optimizely Search & Navigation search index by creating a module dependency class for your search conventions (this where IContent is implemented)

	[ModuleDependency(typeof(IndexingModule))]
	public class FindConventionsInitialization : IInitializableModule
	{
		public void Initialize(InitializationEngine context)
		{
			var client = SearchClient.Instance;
			ContentIndexer.Instance.Conventions.ForInstancesOf<IHasDisableIndex>().ShouldIndex(x => !x.DisableIndex);
		}

		public void Uninitialize(InitializationEngine context) { }
	}

External Search Engines

Use the same boolean property to add instructions for search robots from the <head></head> element in your layout view

<head>
    @if (Model.DisableIndex)
    {
		<meta name="ROBOTS" content="noindex, nofollow" />
    }
</head>

Sitemaps

If you're using the Geta Sitemap generator you can extend it and filter these pages from being added to when the xml file is being generated. Create a class that inherits from the abstract base SitemapXmlGenerator class and interface ICommerceAndStandardSitemapXmlGenerator. Override the AddFilteredContentElement method and from there you can exlude the pages with IDisableIndex.  

    public class CommerceAndStandardSitemapXmlGenerator : SitemapXmlGenerator, ICommerceAndStandardSitemapXmlGenerator
	{
		public CommerceAndStandardSitemapXmlGenerator(
			ISitemapRepository sitemapRepository, 
			IContentRepository contentRepository, 
			UrlResolver urlResolver, 
			ISiteDefinitionRepository siteDefinitionRepository, 
			ILanguageBranchRepository languageBranchRepository, 
			IContentFilter contentFilter) 
			: base(sitemapRepository,  contentRepository, urlResolver, siteDefinitionRepository, languageBranchRepository, contentFilter)
		{
		}

		//Filter content from xml sitemap
		protected override void AddFilteredContentElement(CurrentLanguageContent languageContentInfo, IList<XElement> xmlElements)
		{
			var sitemapContent = languageContentInfo.Content as IHasDisableIndex;

			if (sitemapContent != null && sitemapContent.DisableIndex)
			{
				return;
			}

			base.AddFilteredContentElement(languageContentInfo, xmlElements);
		}
	}
Oct 06, 2021

Comments

Please login to comment.
Latest blogs
Headless forms reloaded (beta)

Forms is used on the vast majority of CMS installations. But using Forms in a headless setup is a bit of pain since the rendering pipeline is based...

MartinOttosen | Mar 1, 2024

Uploading blobs to Optimizely DXP via PowerShell

We had a client moving from an On-Prem v11 Optimizely instance to DXP v12 and we had a lot of blobs (over 40 GB) needing uploading to DXP as a part...

Nick Hamlin | Mar 1, 2024 | Syndicated blog

DbLocalizationProvider v8.0 Released

I’m pleased to announce that Localization Provider v8.0 is finally out.

valdis | Feb 28, 2024 | Syndicated blog

Epinova DXP deployment extension – With Octopus deploy

Example how you can use Epinova DXP deployment extension in Octopus deployment.

Ove Lartelius | Feb 28, 2024 | Syndicated blog