SaaS CMS has officially launched! Learn more now.

Ben Nitti
Oct 6, 2021
(1 votes)

How to exclude pages from your search index, sitemaps and internet search engines

Your client wants to exclude certain pages from showiing up in your site's search results, they also want prevent these pages from being crawled by external search engines (Google, Bing etc.) and lastly they want them removed from their sitemap.xml file.  All of this can be achieved and controlled with a checkbox property on the Settings tab in the editor. I'll demonstrate below. 

Create an interface with a boolean property that inherits IContent

	public interface IDisableIndexing : IContent
		bool DisableIndex { get; set; }

Add it to your base class or any page type 

    public abstract class SitePageData : PageData, IDisableIndex
		[Display(Name = "Disable Indexing",
			Description = "Removes the page from search index, sitemap and search engines",
			GroupName = SystemTabNames.Settings,
			Order = 10)]
		public virtual bool DisableIndex { get; set; }

Optimizely Search & Navigation

You can filter  from your Optimizely Search & Navigation search index by creating a module dependency class for your search conventions (this where IContent is implemented)

	public class FindConventionsInitialization : IInitializableModule
		public void Initialize(InitializationEngine context)
			var client = SearchClient.Instance;
			ContentIndexer.Instance.Conventions.ForInstancesOf<IHasDisableIndex>().ShouldIndex(x => !x.DisableIndex);

		public void Uninitialize(InitializationEngine context) { }

External Search Engines

Use the same boolean property to add instructions for search robots from the <head></head> element in your layout view

    @if (Model.DisableIndex)
		<meta name="ROBOTS" content="noindex, nofollow" />


If you're using the Geta Sitemap generator you can extend it and filter these pages from being added to when the xml file is being generated. Create a class that inherits from the abstract base SitemapXmlGenerator class and interface ICommerceAndStandardSitemapXmlGenerator. Override the AddFilteredContentElement method and from there you can exlude the pages with IDisableIndex.  

    public class CommerceAndStandardSitemapXmlGenerator : SitemapXmlGenerator, ICommerceAndStandardSitemapXmlGenerator
		public CommerceAndStandardSitemapXmlGenerator(
			ISitemapRepository sitemapRepository, 
			IContentRepository contentRepository, 
			UrlResolver urlResolver, 
			ISiteDefinitionRepository siteDefinitionRepository, 
			ILanguageBranchRepository languageBranchRepository, 
			IContentFilter contentFilter) 
			: base(sitemapRepository,  contentRepository, urlResolver, siteDefinitionRepository, languageBranchRepository, contentFilter)

		//Filter content from xml sitemap
		protected override void AddFilteredContentElement(CurrentLanguageContent languageContentInfo, IList<XElement> xmlElements)
			var sitemapContent = languageContentInfo.Content as IHasDisableIndex;

			if (sitemapContent != null && sitemapContent.DisableIndex)

			base.AddFilteredContentElement(languageContentInfo, xmlElements);
Oct 06, 2021


Please login to comment.
Latest blogs
Optimizely release SaaS CMS

Discover the future of content management with Optimizely SaaS CMS. Enjoy seamless updates, reduced costs, and enhanced flexibility for developers...

Andy Blyth | Jul 17, 2024 | Syndicated blog

A day in the life of an Optimizely Developer - London Meetup 2024

Hello and welcome to another instalment of A Day In The Life Of An Optimizely Developer. Last night (11th July 2024) I was excited to have attended...

Graham Carr | Jul 16, 2024

Creating Custom Actors for Optimizely Forms

Optimizely Forms is a powerful tool for creating web forms for various purposes such as registrations, job applications, surveys, etc. By default,...

Nahid | Jul 16, 2024

Optimizely SaaS CMS Concepts and Terminologies

Whether you're a new user of Optimizely CMS or a veteran who have been through the evolution of it, the SaaS CMS is bringing some new concepts and...

Patrick Lam | Jul 15, 2024

How to have a link plugin with extra link id attribute in TinyMce

Introduce Optimizely CMS Editing is using TinyMce for editing rich-text content. We need to use this control a lot in CMS site for kind of WYSWYG...

Binh Nguyen Thi | Jul 13, 2024

Create your first demo site with Optimizely SaaS/Visual Builder

Hello everyone, We are very excited about the launch of our SaaS CMS and the new Visual Builder that comes with it. Since it is the first time you'...

Patrick Lam | Jul 11, 2024