High CPU

[Barry]

Hey team.

Iwonder if someone can help me understand an issue I am seeing. My site (https://allthepics.net) is hosted in a dedicated server (8CPU, 64GB RAM and SSD disks),

The issue is that the CPU is constantly at %100. I notice that if I rename .htaccess to old.htaccess the CPU spike drops (but then obviously the site no longer works).

I don't think this is an issue with Chevereto per see, more than likely there is a loop in a process somewhere, so I was hoping that someone could point me in the right direction.

 

[Rodolfo]

Chevereto includes a single entrypoint guard that prevents calling any .php file on the server. This is a must have security guard.

Most likely they are hitting an unprotected endpoint that causes high CPU load, perhaps a distributed attack exploiting a known service (I had experience this with Facebook botnets). You should implement a network layer bot protection (like CloudFlare) and block the bots that are driving that load.

 

[konj]

Create llms.txt file and upload it in to root of your website with following code inside:

# llms.txt — AI / LLM crawler exclusion
# Purpose: Disallow AI model training, dataset collection, and scraping by known LLM bots

User-agent: GPTBot
Disallow: /

User-agent: OpenAI
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GoogleOther
Disallow: /

User-agent: GoogleOther-Image
Disallow: /

User-agent: Anthropic
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: FacebookBot-LLM
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: AmazonAI
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Applebot-LLM
Disallow: /

User-agent: Applebot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: PerplexityAI
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: CommonCrawl
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: NeevaBot
Disallow: /

User-agent: Sogou
Disallow: /

User-agent: SemrushBot-LLM
Disallow: /

User-agent: AhrefsBot-LLM
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: Baiduspider-LLM
Disallow: /

User-agent: YandexAI
Disallow: /

User-agent: DataForSeoBot-LLM
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: python-requests
Disallow: /

User-agent: curl
Disallow: /

User-agent: wget
Disallow: /

User-agent: *
Allow: /

Add this to your robots.txt as well for redundancy:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Amazonbot
Disallow: /

And as Rodolfo mentioned, I would also add a layer in between, like Cloudflare for example.