All Packages Class Hierarchy This Package Previous Next Index
Class websphinx.RobotExclusion
java.lang.Object
|
+----websphinx.RobotExclusion
- public class RobotExclusion
- extends Object
-
RobotExclusion(String)
- Make a RobotExclusion object.
-
clear()
- Clear the cache of robots.txt entries.
-
disallowed(URL)
- Check whether a URL is disallowed by robots.txt.
-
main(String[])
-
RobotExclusion
public RobotExclusion(String userAgent)
- Make a RobotExclusion object.
- Parameters:
- userAgent - name of the robot using this object, as shown
in the User-Agent header fields of its HTTP requests. Use
null for anonymous robots.
disallowed
public boolean disallowed(URL url)
- Check whether a URL is disallowed by robots.txt.
- Parameters:
- url - URL to test
- Returns:
- true if url's Web site denies robot access to the url
clear
public void clear()
- Clear the cache of robots.txt entries.
main
public static void main(String argv[]) throws Exception
All Packages Class Hierarchy This Package Previous Next Index