• Welcome to the Chevereto user community!

    Here users from all over the world gather around to learn the latest about Chevereto and contribute with ideas to improve the software.

    Please keep in mind:

    • 😌 This community is user driven. Be polite with other users.
    • 👉 Is required to purchase a Chevereto license to participate in this community (doesn't apply to Pre-sales).
    • 💸 Purchase a Pro Subscription to get access to active software support and faster ticket response times.
  • Chevereto Support CLST

    Support response

    Support checklist

Website slowing down on heavy load [Solved]

mkerala

👽 Chevereto Freak
I got over 1M+ images on my site. Now every time someone opens explore it kind of brings down my site. I reduced listing images to 50 from earlier 100. But still CPU is hitting 100% when opening explore. Most of the CPU usage goes to mysql process. Also the site now through Nginx timeout when this happens.

I believe this is due to database query on a large database table. The database size is around 1.3GB now. I am considering moving database to cloud to ease the load on server. But not sure if it is going to solve the issue or would require some database optimization.

Anyone able to provide any expert advise on this would appreciated.
 
Check your mysql settings, lower the number of images per page, etc.
 
I use higher listing images per page for moderation (250/page). Since it is the only way to moderate large no: of images in short time. I am having Maria DB 10.1.25 running default configuration. Please let me know what setting I should change.
 
Issue was with php7.0-fpm. Had below error in php logs

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

edited /etc/php/7.0/fpm/pool.d/www.conf as follows

pm = dynamic
pm.max_children = 100 (original: 5)
pm.start_servers = 20 (original: 1)
pm.min_spare_servers = 20 (original: 1)
pm.max_spare_servers = 30 (original: 3)
pm.max_requests = 1000 (originally commented)


Now able to load 500 images per page without issue. Site also much faster now. CPU and RAM usage too now very low.
 
Last edited:
Sorry, I should have checked the logs more closely. I looked only at Nginx log earlier which showed time out.
 
Then it wasn't MySQL. Glad you fixed it!
Turns out it was only a temporary fix.

I kept on increasing pm.max_children value from 5 to 10, then to 50, 100, 200. Eventually the database started throwing error too many SQL connections.

It seems that php connections to database are waiting too long to get a response. So when I increased pm.max_children it created new connections while the old one still waits. This has also slowed down my database performance to just Queries per second avg: 15.494. Intrestingnly slow queries is always showing zero.

Uptime: 212531 Threads: 1 Questions: 3293126 Slow queries: 0 Opens: 982 Flush tables: 1 Open tables: 976 Queries per second avg: 15.494


I am trying to move the database to a cloud based database service. Hopefully it will improve the performance.
 
What is the update on this? Did moving to cloud-based solution help?
I suggest you perform a database optimization and repair and see if it helps. Sometimes, you may have corrupt database while exporting the database from the previous host and importing that corrupt database into the new host.
 
What is the update on this? Did moving to cloud-based solution help?
I suggest you perform a database optimization and repair and see if it helps. Sometimes, you may have corrupt database while exporting the database from the previous host and importing that corrupt database into the new host.

Exactly some of those issues came when I tried to migrate to Cloud database. OVH is currently looking into it. Once they figure it our hope fully I can move it to cloud soon. My database QPS has droped around 15-16, so I definitely want a powerful machine to run this DB. Also cloud database includes free daily backup.

The reason why I don't want to do much on server because if the CPU hits 100% it will bring down the site. Also it will take a while to do any work on the DB like optimization. Currently high have high traffic coming to my site.
 
At what number did this start slowing down for you guys?

I'm at 610,000 and have not experienced any slowness. Did about 2 terabytes of traffic this month, 0.13 across my 3 CPUs (Load Averages: 0.04 0.19 0.13 as of right now). I loaded up multiple pages in explore, and had them reloading, and my CPU remained within a few points.

Have you buffered your SQL table? In a heavy load environment InnoDB will start to slide down behind MyISAM, because, InnoDB uses a lot of Disk I/O and RAM, and can hit up CPU heavily. For reading content MyISAM is faster. I purposely made sure my images table was MyISAM because it is very rarely edited, unless something new is added, thus, it can be buffered, for quick access later without taxing the server.
 
Mine is over 1 million and I had to switch to external storage when I noticed the slowness. I was planing to step up a separate database server, but it was lot easier and cheaper to get Cloud Database service.

What is the database QPS reported on your dashboard?
 
After trying out Cloud Database service, the QPS wasn't that great at 25-45 range and the service was also running out of memory and getting high slow queries. So I decided to re-configure the server from scratch by splitting all services across 3 virtual server.

1. Nginx webserver with php
2. MariDB database server
3. File storage server with Nginx

Now that I can independently monitor each server and service, I could drill down to the one slowing the site down. I was wrong about database slowing down as the server was barely using using 600MB RAM and it process anything thrown at it in few seconds. Also QPS not much to do with DB performance. Its just average of queries processed in last hour or so. There was absolutely no slow queries reported.

So the slowness was either in Nginx or the PHP module. Nginx seems to be okay as its error log shows timed out waiting for php. I upgraded to latest PHP 7.1 and still no sign of improvement. Its started slowing down on heavy load.

Then I came across below article which advise to use pm = static for max performance and it did the trick. The default PHP-FPM settings are designed to use the least amount of RAM more suitable for shared hosting environment. My server RAM usage was barely 700MB out of 4GB available even under high load. I set pm.max_children= 100 and set pm=static. The RAM usage spiked to 1.7GB and website is responding faster than ever. Even with listing set to 500 the Explore loads in a few seconds.

Reference: https://haydenjames.io/php-fpm-tuning-using-pm-static-max-performance/
 
I did not know you had it set to dynamic otherwise I would have said to change it to static. I always use custom set of pool in php for each website on my server and set pm= static and max children accordingly.

Good that you figured it out. :)
 
Last edited:
Yeah, it's important to mention that max children must reflect the amount of ram you have. If you have 8gb you should set it to no higher than 72 and static. This way your server is on edge but also able to breathe under heavy load.
 
Back
Top