Deane Barker
Sep 21, 2010
(2 votes)

A Simple ROBOTS.TXT Handler in EPiServer

We’re moving a client from a Dreamweaver-based site to an EPiServer site.  One of the only reasons they had left to FTP into the site at any time was to manage the robots.txt file.  We wanted to eliminate that reason so we could shut FTP off completely.

This is what we did:

1. Added a Long String property on the Start Page called “RobotsTxt”.  We dumped the contents of the existing robots.txt in there.

2. Add this line to our web.config:

<add name="RobotsTxtHandler" preCondition="integratedMode" verb="*" path="robots.txt" type="Blend.EPiServer.RobotsTxtHandler, [Assembly Name]" />

3. Compiled this class into our project:

using System.Web;
using EPiServer;
using EPiServer.Core;

namespace Blend.EPiServer
    public class RobotsTxtHandler : IHttpHandler
        public bool IsReusable { get { return true; } }

        public void ProcessRequest(HttpContext context)
            context.Response.ContentType = "text/plain";

Now, the client edits their robots.txt text in EPiServer. Calling “/robots.txt” will dump the contents of that Start Page property into the output as plain text.

(The larger principle at work here is that the Start Page is generally treated as a Singleton in EPiServer – there will only ever be one per site.  This means we tend to load it up with global properties – data you just need to store and manage in EPiServer, but will never publish as its own page.  Things like this robots.txt text, the TITLE tag suffix for the site, random Master Page text strings, etc.)

Sep 21, 2010


Please login to comment.
Latest blogs
Implementing EmbeddedLocalization in Optimizely CMS 12

My previous post on translation (Translating Optimizely CMS 12 UI components) gives an overview of how to implement the FileXmlLocalizationProvider...

Eric Herlitz | Jan 27, 2023 | Syndicated blog

Breaking changes in EPiServer.CMS.TinyMce 4.0.0

After upgrading to the latest version of EPiServer.CMS.TinyMce, the dropdown with formats disappears. Learn how to get it back!

Tomas Hensrud Gulla | Jan 27, 2023 | Syndicated blog

Translating Optimizely CMS 12 UI components

Optimizely CMS 12 have been out for a while now, but still some elements haven't been properly translated resulting in a GUI defaulting to english....

Eric Herlitz | Jan 26, 2023 | Syndicated blog

Image preview in Optimizely CMS12 all properties view

With these simple steps, you can now see an Image and its Metadata, including size and dimensions, when editing an Image property in Optimizely...

Tomas Hensrud Gulla | Jan 26, 2023 | Syndicated blog