robots

    Top: Computers: Internet: Searching: Search_Engines: Robots:

• Top/Computers/Software/Internet/Site Management/Log Analysis
• Top/Computers/Internet/Web Design and Development/Authoring/Online Tools/robots.txt
• Top/Computers/Internet/Abuse/Spam/Preventing

• - An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
  
• - Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.
  
• - John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
  
• - Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
  
• - Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
  
• - A list from PGTS of Web robots with the identifying data they leave in Web site logs.
  
• - José Luis Pellicer's searchable database of robots, spiders and other user agents for programs that surf the web.