Concept of Robots.txt

Concept of Robots.txt

The Robots.txt is a simple file containing text matter and is always placed/applied at the root directory only. The main function of Robots.txt is to tell various Search Engines as well as other Robots that which pages are to visit & index and which are to leave.

Example

www.example.com/robots.txt

Classification of Robots.txt

Robots.txt is classified into 3 major parts as listed below:-

1. Websites or Directories (Robots.txt)
2. Single Page (Robots Metatag)
3. Single Link or URL (Nofollow Attribute)

Syntax

User-agent: *

Disallow: /

<meta name="robots" content="noindex,nofollow">
or

<meta name="robots" content="none">

<a href=http://www.example.com/ rel="nofollow">Example</a>

Note

For more concepts about Robots.txt, visit http://www.mcanerin.com/.