Robots.txt

[mariov]

Wich code for robots.txt is the best for cherevato

 

[Rodolfo]

The usual "crawl all" robots.txt should be enough.

 

[mariov]

User-agent: *
This is innaf you meen?

 

[Rodolfo]

This:

User-agent: *
Disallow:

 

[mariov]

Thanks

 

[Andy]

This:

User-agent: *
Disallow:

Mine is:

User-Agent: *
Allow: /

Is it same? Or which one is preferred?

 

[Rodolfo]

You don't need a robots.txt if you will allow robots to index everything. You only need a robots.txt directive when you want to tweak specific paths and routes.

 

[Andy]

You don't need a robots.txt if you will allow robots to index everything. You only need a robots.txt directive when you want to tweak specific paths and routes.

The only reason I'm putting robots.txt is because my httpd.log is flooded with missing robots.txt file.