Robots.txt Generator
Robots.txt Tool from HowRank.com creates your robots.txt file for you. You can even include your SiteMap for better indexing.
Googlebot is a search bot used by Google. It collects documents from the web to build a searchable index for the Google search engine. (cited: Wikipedia) [ok back to robots.txt tool]
Yahoo! Slurp is a web crawler from Yahoo! group. It crawls the web and puts content into the Yahoo! Search engine. Slurp is based on Inktomi's web search technology - Inktomi was acquired by Yahoo! in late 2002. (cited: Wikipedia) [ok back to robots.txt tool]
MSNbot is a web-crawling robot (type of internet bot), deployed by Microsoft to supply Live Search. It collects documents from the web to build a searchable index for the MSN Search engine, which went into beta in 2004, and had full public release in 2005. It performs the same function as Google's Googlebot, and Yahoo!'s slurp. (cited: Wikipedia) [ok back to robots.txt tool]
Ask.com has its own bot as well. Teoma is its name. It works like the others. [ok back to robots.txt tool]
Hide Folders If you have a folder you don't want everyone else in the world to see for some reason or another. Maybe it contains personal data? Then you can effectively "rope off" that area of your site from bots by adding the folder name. As an example say we have a folder called personal, and we don't want the bots to index it. We would then add /personal/ in one of the four boxes and it will block that area off when you create your fie. [ok back to robots.txt tool]
Need robot parts? CircuitBoardFix.com will repair or replace any defective robot parts you need. Please visit their site and either call or email them.
