• Welcome to the Chevereto user community!

    Here users from all over the world gather around to learn the latest about Chevereto and contribute with ideas to improve the software.

    Please keep in mind:

    • 😌 This community is user driven. Be polite with other users.
    • 👉 Is required to purchase a Chevereto license to participate in this community (doesn't apply to Pre-sales).
    • 💸 Purchase a Pro Subscription to get access to active software support and faster ticket response times.

Disabling server indexing

A

Azone

Chevereto Member
Hello, how can I disable Chevereto indexing so that the site does not appear in the search engine when requesting a domain, robots.txt does not help, does anyone know how to disable this thing so that the site disappears from search engines?
I use a translator
 
Solution
Rodolfo
Azone said:
robots.txt does not help
Click to expand...
The robots.txt file works for crawlers which respect your opinion about data crawling, like Google. But there are several crawlers which won't follow any of these rules.

The best you can do to solve this problem is to add firewall rules to make the website not accessible to the parties you don't want to welcome.

For example, Facebook loves to crawl Chevereto's demo content which is a problem for me because they took all the resources and deny service to other users. As I don't like that I simply blocked their ASN.
Sort by date Sort by votes
Azone said:
robots.txt does not help
Click to expand...
The robots.txt file works for crawlers which respect your opinion about data crawling, like Google. But there are several crawlers which won't follow any of these rules.

The best you can do to solve this problem is to add firewall rules to make the website not accessible to the parties you don't want to welcome.

For example, Facebook loves to crawl Chevereto's demo content which is a problem for me because they took all the resources and deny service to other users. As I don't like that I simply blocked their ASN.
 
Upvote 1 Downvote
Solution
interesting.. i was wondering why i kept seeing these weird facebook ips on my statcounter statistics

these facebook bots crawl my site but only the chevereto site and not my xenforo forum.

I wonder why they like crawling chevereto so much.🤔

facebook-crawl.png
 
Upvote 0 Downvote
@mar In some cases it is Facebook crawling (usual stuff) but at times it could be someone exploiting Facebook to trigger denial of service. The demo has been attacked several times by these, but the demo remains.
 
Upvote 0 Downvote
mar said:
interesting.. i was wondering why i kept seeing these weird facebook ips on my statcounter statistics

these facebook bots crawl my site but only the chevereto site and not my xenforo forum.

I wonder why they like crawling chevereto so much.🤔

View attachment 6192
Click to expand...
Same on my side there is a lot of crawler (Facebook & GPTBot and much more) with an insane amout of http request (only on my chevereto website and not my main blog for example).

1730803172308.png

I blocked them on my CloudFlare WAF directly.
 
Upvote 0 Downvote
You must log in or register to reply here.
Back
Top