SaaS CMS has officially launched! Learn more now.

Apr 27, 2015
(4 votes)

Simple robots.txt built with mvc

The other day there was a question about serving static files from episerver, like a robots.txt. Since most editor like to be able to update the robots.txt them self many of us has build solutions for that. There are also some packages in the nuget feed from EPiServer that will help you do just that.

But if you are like me, I do not like to have pre-compiled solutions from other companies in my project, some are better then others but still I like to be able to change most of my code in a project. Of course you can download the source code in many of those packages but some times they tend to do a bit more then you might expected as well.


In this case I just would like to serve a robots.txt if it is requested by a search engine for instance.


First I create the robots.txt controller. Since we are working with EPiServer CMS I have added a textarea on the startpage type and therefore editors are able to change the content of the file.


   1:      public class RobotsTxtController : Controller
   2:      {
   3:          private Injected<IContentLoader> ContentLoader { get; set; }
   5:          [ContentOutputCache]
   6:          public ActionResult Index()
   7:          {
   8:              var startPage = ContentLoader.Service.Get<StartPage>(ContentReference.StartPage);
  10:              string content = startPage.RobotsTxtContent;
  11:              return Content(content, "text/plain");
  12:          }
  13:      }


Nesx step is to create an initialization module were we map a route to our robots.txt file.


   1:      [InitializableModule]
   2:      [ModuleDependency(typeof(EPiServer.Web.InitializationModule))]
   3:      public class RenderingInitialization : IInitializableModule
   4:      {
   5:          public void Initialize(InitializationEngine context)
   6:          {
   8:              //Maps route for robots.txt
   9:              RouteTable.Routes.MapRoute("RobotsTxtRoute", "robots.txt", new { controller = "RobotsTxt", action = "Index" });
  10:          }
  13:          public void Preload(string[] parameters) { }
  15:          public void Uninitialize(InitializationEngine context)
  16:          {
  17:              //Add uninitialization logic
  19:          }
  20:      }


And finally we add the property to our startpage model

        [Display(Name = "Robots.txt", Order = 90, GroupName = Global.GroupNames.SiteSettings)]
        public virtual string RobotsTxtContent { get; set; }
Apr 27, 2015


Marija Jemuovic
Marija Jemuovic Apr 27, 2015 09:57 AM

Nice idea!

Eric Apr 27, 2015 04:26 PM


valdis Apr 28, 2015 12:30 PM

No offense, just wanted to know reason using Injected..? :)

Vincent Apr 29, 2015 06:57 AM

To Valdis

Is it used for querying the pagedata from startpage reference?

Feb 14, 2017 06:01 PM

Thanks Eric, nice & simple => a perfect solution.

Eric Feb 15, 2017 09:02 AM

Happy you like it :) 

Please login to comment.
Latest blogs
Optimizely London Dev Meetup 11th July 2024

On 11th July 2024 in London Niteco and Netcel along with Optimizely ran the London Developer meetup. There was an great agenda of talks that we put...

Scott Reed | Jul 19, 2024

Optimizely release SaaS CMS

Discover the future of content management with Optimizely SaaS CMS. Enjoy seamless updates, reduced costs, and enhanced flexibility for developers...

Andy Blyth | Jul 17, 2024 | Syndicated blog

A day in the life of an Optimizely Developer - London Meetup 2024

Hello and welcome to another instalment of A Day In The Life Of An Optimizely Developer. Last night (11th July 2024) I was excited to have attended...

Graham Carr | Jul 16, 2024

Creating Custom Actors for Optimizely Forms

Optimizely Forms is a powerful tool for creating web forms for various purposes such as registrations, job applications, surveys, etc. By default,...

Nahid | Jul 16, 2024

Optimizely SaaS CMS Concepts and Terminologies

Whether you're a new user of Optimizely CMS or a veteran who have been through the evolution of it, the SaaS CMS is bringing some new concepts and...

Patrick Lam | Jul 15, 2024

How to have a link plugin with extra link id attribute in TinyMce

Introduce Optimizely CMS Editing is using TinyMce for editing rich-text content. We need to use this control a lot in CMS site for kind of WYSWYG...

Binh Nguyen Thi | Jul 13, 2024