Try our conversational search powered by Generative AI!

Deane Barker
Sep 21, 2010
(2 votes)

A Simple ROBOTS.TXT Handler in EPiServer

We’re moving a client from a Dreamweaver-based site to an EPiServer site.  One of the only reasons they had left to FTP into the site at any time was to manage the robots.txt file.  We wanted to eliminate that reason so we could shut FTP off completely.

This is what we did:

1. Added a Long String property on the Start Page called “RobotsTxt”.  We dumped the contents of the existing robots.txt in there.

2. Add this line to our web.config:

<add name="RobotsTxtHandler" preCondition="integratedMode" verb="*" path="robots.txt" type="Blend.EPiServer.RobotsTxtHandler, [Assembly Name]" />

3. Compiled this class into our project:

using System.Web;
using EPiServer;
using EPiServer.Core;

namespace Blend.EPiServer
    public class RobotsTxtHandler : IHttpHandler
        public bool IsReusable { get { return true; } }

        public void ProcessRequest(HttpContext context)
            context.Response.ContentType = "text/plain";

Now, the client edits their robots.txt text in EPiServer. Calling “/robots.txt” will dump the contents of that Start Page property into the output as plain text.

(The larger principle at work here is that the Start Page is generally treated as a Singleton in EPiServer – there will only ever be one per site.  This means we tend to load it up with global properties – data you just need to store and manage in EPiServer, but will never publish as its own page.  Things like this robots.txt text, the TITLE tag suffix for the site, random Master Page text strings, etc.)

Sep 21, 2010


Please login to comment.
Latest blogs
Optimizely Web... 6 Game Changing Features in 2024

If you are interested in learning about what's new within Optimizely Web, you are in the right place. Carry on reading to learn about the 6 greates...

Jon Jones | Mar 3, 2024 | Syndicated blog

Headless forms reloaded (beta)

Forms is used on the vast majority of CMS installations. But using Forms in a headless setup is a bit of pain since the rendering pipeline is based...

MartinOttosen | Mar 1, 2024

Uploading blobs to Optimizely DXP via PowerShell

We had a client moving from an On-Prem v11 Optimizely instance to DXP v12 and we had a lot of blobs (over 40 GB) needing uploading to DXP as a part...

Nick Hamlin | Mar 1, 2024 | Syndicated blog

DbLocalizationProvider v8.0 Released

I’m pleased to announce that Localization Provider v8.0 is finally out.

valdis | Feb 28, 2024 | Syndicated blog

Epinova DXP deployment extension – With Octopus deploy

Example how you can use Epinova DXP deployment extension in Octopus deployment.

Ove Lartelius | Feb 28, 2024 | Syndicated blog

Identify Azure web app instance id's for an Optimizely CMS site

When running Optimizely CMS in Azure, you will be using an instance bound cloud license. What instances are counted, and how can you check them? Le...

Tomas Hensrud Gulla | Feb 27, 2024 | Syndicated blog