Deane Barker
Sep 21, 2010
visibility 11492
star star star star star
(2 votes)

A Simple ROBOTS.TXT Handler in EPiServer

We’re moving a client from a Dreamweaver-based site to an EPiServer site.  One of the only reasons they had left to FTP into the site at any time was to manage the robots.txt file.  We wanted to eliminate that reason so we could shut FTP off completely.

This is what we did:

1. Added a Long String property on the Start Page called “RobotsTxt”.  We dumped the contents of the existing robots.txt in there.

2. Add this line to our web.config:

<add name="RobotsTxtHandler" preCondition="integratedMode" verb="*" path="robots.txt" type="Blend.EPiServer.RobotsTxtHandler, [Assembly Name]" />

3. Compiled this class into our project:

using System.Web;
using EPiServer;
using EPiServer.Core;

namespace Blend.EPiServer
{
    public class RobotsTxtHandler : IHttpHandler
    {
        public bool IsReusable { get { return true; } }

        public void ProcessRequest(HttpContext context)
        {
            context.Response.ContentType = "text/plain";
            context.Response.Write(DataFactory.Instance.GetPage(PageReference.StartPage).Property["RobotsTxt"].ToString());
        }
    }
}

Now, the client edits their robots.txt text in EPiServer. Calling “/robots.txt” will dump the contents of that Start Page property into the output as plain text.

(The larger principle at work here is that the Start Page is generally treated as a Singleton in EPiServer – there will only ever be one per site.  This means we tend to load it up with global properties – data you just need to store and manage in EPiServer, but will never publish as its own page.  Things like this robots.txt text, the TITLE tag suffix for the site, random Master Page text strings, etc.)

Sep 21, 2010

Comments

error Please login to comment.
Latest blogs
Pick images from Optimizely DAM straight into rich text

Rich text editors in Optimizely CMS (SaaS), and in CMS 13.7.0 on sites that use the Content Manager, have a new image button. It puts images from t...

Stuart Greig | Nov 18, 2026

Setting up the TinyMCE image picker, and DAM images in Graph and the CMS REST API

The new TinyMCE image and video pickers put images and video from the CMS and from Optimizely DAM straight into rich text, in Optimizely CMS (SaaS)...

Stuart Greig | Nov 18, 2026

Add your own image source to the TinyMCE image picker

In CMS 13.7.0,  the TinyMCE image picker lets editors browse images from the CMS and from Optimizely DAM. Images from your own library can also be...

Stuart Greig | Nov 18, 2026

Lessons From the Optimizely Web Experimentation Compromise

Introduction On 16th November 2026, Scott Helme and the team at Report URI published a detailed investigation into malicious JavaScript being serve...

Mark Stott | Nov 17, 2026