About Robot TXT

What is Robot TXT? Why should we?

BOT also Web Robots, Crawlers and Spiders are known as. Like Google, Yahoo MSN search engine to index web content as they use; spammers use email addresses to scan them, and other uses.

Web pages and content of your website guidelines robot robot.txt robot to determine the index file. Spiders after files, so that provides a map of what they can and can not index. This file will be available in your website root list. URL of your robots.txt file path to the web address / name must be as robots.txt.

How to create robots.txt file?

User-agent: *

Asterisk (*) or wildcard represents a special value and any robotic device. Asterisk (* agent) or a wildcard field robot user "and therefore need only one until you fully understand how to set up separate user - means an agent.

Disclaimer:

Disclaimer: Not a line / (Forward Slash) robots (crawlers) that they can index a website that reflects all webpages.

A blank indicates that all URLs prices can bring. At least one rejection region is / (forward slash should be without a record) as shown above.

Disclaimer: / private /

Bots can not do that this material shows that the index of / private / directory.

Syntax to create robots.txt files

Allow all incoming full Robots:

User-agent: *

Disclaimer:

Server for all Robots by:

User-agent: *

Disclaimer: /

All parts of the robot from a server outside:

User-agent: *

Disclaimer: / private /

Disclaimer: images / - protected /

Disclaimer: images / work /

For a robot from the server:

User-agent: BOT Name

Disclaimer: /

In some parts of a robot from a server:

User-agent: BOT Name

Disclaimer: / private /

Disclaimer: images / - protected /

Disclaimer: images / work /

If a particular file in the folder that you want to refuse:

Disclaimer: / / for nothing-Private display.htm

Note that using the example above a specified page (for anything-display.htm) is not included, but outside / private / list will not serve.

An empty "/ robots.txt file is useless presence, treat as if it was not present, then all robots will be invited will be considered automatically.

You need to show your robots.txt file. Enter the full URI robots.txt file on your server. Robots.txt files is always in your web root.