November Happy Hour will be moved to Thursday December 5th.

AI OnAI Off

Robots.txt for different sites

Chandra Shekar
Chandra Shekar
Vote:
 

Hi All,

I have a EPi server web application. The same web application is hosted under 3 domains.  Now my requirement is to have diffrenet Robots.txt for different domains. But since all the domains are pointing to sane folder I can have only file placed in that folder. Is there any workaround for placing seperate Robots.txt file under each domain.

 

Thanks in Advance.

#33163
Oct 03, 2009 11:03
Maras M
Maras M
Vote:
 

You can create 3 files:

robotsDomain1.txt 

robotsDomain2.txt 

robotsDomain3.txt  

and then define your own handler, assign it to robots.txt in web.config

<httpHandlers>
<add verb="*" path="robots.txt" type="Project.RobotsHandler"/>
</httpHandlers>

and inside the handler check Request.Url.DnsSafeHost and return the content of one of the files.  If you use IIS7 you define handlers in section

<system.webServer>
<handlers>
<add name="robots" path="robots.txt" verb="*" type="Project.RobotsHandler, Project" />
</handlers>
</system.webServer>
 

#33165
Oct 03, 2009 11:14
Chandra Shekar
Chandra Shekar
Vote:
 

Thanks for your reply.

Can you point me to any URL that explains the code to be written in the Handler.

#33166
Oct 03, 2009 11:39
Maras M
Maras M
Vote:
 
http://www.kleenecode.net/2007/11/17/dynamic-robotstxt-with-aspnet-20/
#33167
Oct 03, 2009 12:11
Chandra Shekar
Chandra Shekar
Vote:
 

Thanks Maras. Below link is also usefull

http://www.atlasbay.com/2008/11/06/robotstxt-httphandler-in-iis7/ 

Thanks for your suggestion.

 

#33168
Oct 03, 2009 12:25
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.