Robot.txt

[Youssef]

Hi Then I have a robot.txt file so that google does not index the system files. Because google asks for it. Thank you

 

[Youssef]

Can someone help me please.

 

[Rodolfo]

Robots.txt is not mandatory. Anyway, check here: http://tools.seobook.com/robots-txt/generator/

 

[Jailer]

Just because you have a robots.txt file doesn't mean they will follow it.

http://www.robotstxt.org/robotstxt.html

Anyway if you still want one create a text file named robots.txt in your web root and insert the following code:

User-agent: *
Disallow: /

 

[Youssef]

Yes okay for your answer but what I wanted to know was what chevereto files to ban. Thank you

 

[Rodolfo]

Yes okay for your answer but what I wanted to know was what chevereto files to ban. Thank you

¯\_(ツ)_/¯

Up to you which paths you ban.

 

[JakeSully]

if you do following code like jailer said, you can write the path after / part example

User-agent: *
Disallow: /
Disallow: /images
Disallow: /albums
Disallow: /users

and so on. / means root path and with that one only it will block everything within root location.

 

[Oakley]

But that would mean nothing gets indexed ??

Disallow: / would disallow anything below root so the last 3 lines are superflous.

Google cannot index the system files because it has no access to them anyway.

 

[friv]

How look like robots.txt file which will allow to index all pages? So i do not want to block crawlers for my website. I want to allow all searchengines to crawl my website.

 

[JakeSully]

Then you change disallow: / to Allow: /

 

[friv]

Then you change disallow: / to Allow: /

What will happen if i have only this in robots.txt ?

User-agent: *

 

[imghut]

How look like robots.txt file which will allow to index all pages? So i do not want to block crawlers for my website. I want to allow all searchengines to crawl my website.

Then don't use a robot.txt file 😉

 

[friv]

Then don't use a robot.txt file 😉

Sorry what you mean by that? This is very important for me. Can you explain?

 

[imghut]

If you want all crawlers on your site you do NOT need a robots.txt file, just delete it.

 

[friv]

If you want all crawlers on your site you do NOT need a robots.txt file, just delete it.

Thanks.

I was have all time in robots.txt file this : User-agent: *

So maybe i was blocked everything. OMG

 

[JakeSully]

Thanks.

I was have all time in robots.txt file this : User-agent: *

So maybe i was blocked everything. OMG

no User-agent: * means all bots/agent's is counted, so it does not block as long you do not have Disallow: / after it.

 

[friv]

no User-agent: * means all bots/agent's is counted, so it does not block as long you do not have Disallow: / after it.

Yes,that is clear.