Robots

Robots.txt

Disallow: /
disallow robots to read and search all file after the first "/"
often say every thing after your domain or
after your IP

Disallow:
It is empty after "Disallow:" then it means allow

---------- attention -----------
not all the r

阻止搜索引擎漫游器读取网站第一个"/"
之后的所有网页或特定目录

Robots.txt shall be added at root, no other places
ex: http://----.com/robots.txt

User-Agent:*
Disallow: /
( disallow all robots to search all files)
------------- attention -----------

if you want allow all search engines
then suggest you :
way 1: To delete robots.txt because you do not need a robots.txt
way 2: To keep robots.txt BLANK so no errors fill in error log 
when bots come.
Way 3: no robots.txt but create a customized error page

Do not recommend you to use 
User-Agent:*
Disallow:
to allow all robots to read your files

Because not all robots follow the same rules
the better way to allow all robots to read your all sites
which is

Way 1: " to delete your robots.txt "
But without robots.txt will create errors when bots come.

Way 2: "to keep robots.txt with nothing in" 
so you can avoid errors when bots come.

Way 3: no robots.txt but create a customized error page

Big Tips: read robots line by line each time

Robots.txt 是一个标准档案，其功能是不让
Googlebot 从您的 Web 服务器下载数据。
同时也适用在其它一些主要的搜索引擎漫游器。