Information for Site Administrators

User-Agent: Attribot-Feeds

Visits feeds at most once per hour. If excluded by robots.txt, entries will not appear on public pages, unless they are reviewed or annotated. Even if excluded, feeds will be checked by this agent; entries are displayed for private use of users of the Attribyte console (a private application similar to Google Reader). If this agent is explicitly excluded, for example:

User-Agent: Attribot-Feeds
Disallow: /

feeds will not be checked at all, preventing their entries from being read by anyone using the Attribyte feed-reader. Robots.txt will continue be checked once per day, even if Attribyte-Feeds is excluded.

User-Agent: Attribot-Images

Downloads favicons and creates thumbnail images from images discovered in feed entries that meet specific criteria (dimensions) unless excluded by robots.txt. Small image thumbnails are saved for recent entries, then automatically deleted as the entries age.

User-Agent: Attribot-Pages

Extracts information from HTML pages unless excluded by robots.txt.

Bandwidth Minimization

Attribyte's software agents try to minimize consumed bandwidth through the following mechanisms:

• All requests pass through a caching proxy.
• The If-Modified-Since header is used.
• HEAD is used to dereference links when the content is not important.
• If Content-Encoding gzip is supported by the server, gzip encoding is used for download.
• Robots.txt is cached for 24 hours.