Try our conversational search powered by Generative AI!

Deane Barker
Sep 21, 2010
(2 votes)

A Simple ROBOTS.TXT Handler in EPiServer

We’re moving a client from a Dreamweaver-based site to an EPiServer site.  One of the only reasons they had left to FTP into the site at any time was to manage the robots.txt file.  We wanted to eliminate that reason so we could shut FTP off completely.

This is what we did:

1. Added a Long String property on the Start Page called “RobotsTxt”.  We dumped the contents of the existing robots.txt in there.

2. Add this line to our web.config:

<add name="RobotsTxtHandler" preCondition="integratedMode" verb="*" path="robots.txt" type="Blend.EPiServer.RobotsTxtHandler, [Assembly Name]" />

3. Compiled this class into our project:

using System.Web;
using EPiServer;
using EPiServer.Core;

namespace Blend.EPiServer
    public class RobotsTxtHandler : IHttpHandler
        public bool IsReusable { get { return true; } }

        public void ProcessRequest(HttpContext context)
            context.Response.ContentType = "text/plain";

Now, the client edits their robots.txt text in EPiServer. Calling “/robots.txt” will dump the contents of that Start Page property into the output as plain text.

(The larger principle at work here is that the Start Page is generally treated as a Singleton in EPiServer – there will only ever be one per site.  This means we tend to load it up with global properties – data you just need to store and manage in EPiServer, but will never publish as its own page.  Things like this robots.txt text, the TITLE tag suffix for the site, random Master Page text strings, etc.)

Sep 21, 2010


Please login to comment.
Latest blogs
Optimizely and the never-ending story of the missing globe!

I've worked with Optimizely CMS for 14 years, and there are two things I'm obsessed with: Link validation and the globe that keeps disappearing on...

Tomas Hensrud Gulla | Apr 18, 2024 | Syndicated blog

Visitor Groups Usage Report For Optimizely CMS 12

This add-on offers detailed information on how visitor groups are used and how effective they are within Optimizely CMS. Editors can monitor and...

Adnan Zameer | Apr 18, 2024 | Syndicated blog

Azure AI Language – Abstractive Summarisation in Optimizely CMS

In this article, I show how the abstraction summarisation feature provided by the Azure AI Language platform, can be used within Optimizely CMS to...

Anil Patel | Apr 18, 2024 | Syndicated blog

Fix your Search & Navigation (Find) indexing job, please

Once upon a time, a colleague asked me to look into a customer database with weird spikes in database log usage. (You might start to wonder why I a...

Quan Mai | Apr 17, 2024 | Syndicated blog