...
| Info | ||
|---|---|---|
| ||
A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. |
5.2.2. Strategy
It is decided to make a distinction between two 'layers' of robot filtering (see also figure 5):
...
- The current version of the list can be reached by placing /current/ in the local path of the URI, e.g.: http://purl.org/robotslist/current/robotlist.xml
- An overview of the previous versions can be found by going to the parent of the /current/ localpath element, e.g.: http://purl.org/robotslist/
- Previous versions of the robot list can referred to by using the preferred date instead of the /current/ local path element, e.g.: http://purl.org/robotslist/2010/05/12/robotlist.xml
...