Nearly every site has one, they tell search engine spiders and other web robots where they should and shouldn't visit on the site. They can be extremely complex, with different rules depending upon the robot and the area of the site or they can be as simple as two lines long. Learn more about how web robots use the robots.txt file and how to create one that does what you want it to.

Web Robots - What Are They and How to Use Them

• Sample robots.txt Files
• What does Your Robots.txt File Look Like?
• What is a Web Robot? How to Handle Robots on Your Site and Server
• How to Write a robots.txt File
• Controlling Web Robots

Curious About Robots.txt Files? originally appeared on About.com Web Design / HTML on Tuesday, November 2nd, 2010 at 18:40:13.

Permalink | Comment | Email this