Five New Optimizely Certifications are Here! Validate your expertise and advance your career with our latest certification exams. Click here to find out more

AI OnAI Off

Class RobotsTxtParser

Inheritance
System.Object
RobotsTxtParser
Namespace: EPiServer.LinkAnalyzer.Robots.Internal
Assembly: EPiServer.LinkAnalyzer.dll
Version: 12.0.3
Syntax
public class RobotsTxtParser : Object

Constructors

RobotsTxtParser(String, IWebProxy)

Declaration
public RobotsTxtParser(string userAgent, IWebProxy proxy)
Parameters
Type Name Description
System.String userAgent
System.Net.IWebProxy proxy

Methods

LoadAndParse(Uri)

Declaration
public RobotsEntry LoadAndParse(Uri url)
Parameters
Type Name Description
System.Uri url
Returns
Type Description
RobotsEntry

ParseRobotsTxt(String)

Declaration
public RobotsEntry ParseRobotsTxt(string content)
Parameters
Type Name Description
System.String content
Returns
Type Description
RobotsEntry

Extension Methods