Try our conversational search powered by Generative AI!

Deane Barker
Sep 21, 2010
(2 votes)

A Simple ROBOTS.TXT Handler in EPiServer

We’re moving a client from a Dreamweaver-based site to an EPiServer site.  One of the only reasons they had left to FTP into the site at any time was to manage the robots.txt file.  We wanted to eliminate that reason so we could shut FTP off completely.

This is what we did:

1. Added a Long String property on the Start Page called “RobotsTxt”.  We dumped the contents of the existing robots.txt in there.

2. Add this line to our web.config:

<add name="RobotsTxtHandler" preCondition="integratedMode" verb="*" path="robots.txt" type="Blend.EPiServer.RobotsTxtHandler, [Assembly Name]" />

3. Compiled this class into our project:

using System.Web;
using EPiServer;
using EPiServer.Core;

namespace Blend.EPiServer
    public class RobotsTxtHandler : IHttpHandler
        public bool IsReusable { get { return true; } }

        public void ProcessRequest(HttpContext context)
            context.Response.ContentType = "text/plain";

Now, the client edits their robots.txt text in EPiServer. Calling “/robots.txt” will dump the contents of that Start Page property into the output as plain text.

(The larger principle at work here is that the Start Page is generally treated as a Singleton in EPiServer – there will only ever be one per site.  This means we tend to load it up with global properties – data you just need to store and manage in EPiServer, but will never publish as its own page.  Things like this robots.txt text, the TITLE tag suffix for the site, random Master Page text strings, etc.)

Sep 21, 2010


Please login to comment.
Latest blogs
Create your first demo site with Optimizely SaaS/Visual Builder

Hello everyone, We are very excited about the launch of our SaaS CMS and the new Visual Builder that comes with it. Since it is the first time you'...

Patrick Lam | Jul 11, 2024

Integrate a CMP workflow step with CMS

As you might know Optimizely has an integration where you can create and edit pages in the CMS directly from the CMP. One of the benefits of this i...

Marcus Hoffmann | Jul 10, 2024

GetNextSegment with empty Remaining causing fuzzes

Optimizely CMS offers you to create partial routers. This concept allows you display content differently depending on the routed content in the URL...

David Drouin-Prince | Jul 8, 2024 | Syndicated blog

Product Listing Page - using Graph

Optimizely Graph makes it possible to query your data in an advanced way, by using GraphQL. Querying data, using facets and search phrases, is very...

Jonas Bergqvist | Jul 5, 2024