Child pages
  • KE Usage Statistics Guidelines

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info
titleDefinition of a "robot" according to robotstxt.org

A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.
- http://www.robotstxt.org/faq/what.html, also used as definition by (Geens, 2006), (Heinonen, 1996})

5.2.2. Strategy

It is decided to make a distinction between two 'layers' of robot filtering (see also figure 5):

...

...