Author Topic: The Web Robots Pages can teach you more about web robots - the Good Ones  (Read 946 times)

0 Members and 1 Guest are viewing this topic.

Software Santa

  • Administrator
  • *****
  • Posts: 4280
  • OS:
  • Mac OS X 10.9 Mac OS X 10.9
  • Browser:
  • SeaMonkey 2.30 SeaMonkey 2.30
The Web Robots Pages can teach you more about web robots - the Good Ones. "Bad" Robots ignore the Robots Page on your site ... or else they go straight to the folders that it SAYS they are not allowed in.

Quote

The Web Robots Pages


 Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.


On this site you can learn more about web robots.


 
About robotstxt.org


History
The Web Robot Pages is an information resource dedicated to web robots. Initially hosted at WebCrawler in 1995, it moved to this dedicated site hosted by independent robotstxt.org in 2000. It underwent a modernisation in 2007.


Advertising
At this time we do not offer advertising opportunities to new partners, nor are we interested in selling the domain.


Contact
To contact the administrators of this site regarding technical issues relating to the operation of this site only, please use the contact page.


 
Tools
 
 

 



Other Sites


Google


Many people end up on this site because they have questions about specific search engine robots and search engines. For such questions the best place is the relevant's site's own help pages:



Extensions to the Robots Exclusion Protocol


Recently three major search engines have collaborated to support extensions to the /robotst.txt directives and related mechanisms. See the join announcements on:



http://www.robotstxt.org/

 

email