• Welcome to the Chevereto user community!

    Here users from all over the world gather around to learn the latest about Chevereto and contribute with ideas to improve the software.

    Please keep in mind:

    • 😌 This community is user driven. Be polite with other users.
    • 👉 Is required to purchase a Chevereto license to participate in this community (doesn't apply to Pre-sales).
    • 💸 Purchase a Pro Subscription to get access to active software support and faster ticket response times.

Stuck on 3.8.3 – unable to update

Status
Not open for further replies.

gobzer

Chevereto Member
12M Images 37,564 Albums 992,654 Users 1.9 TB Disk used
Chevereto version 3.8.3
Tried to update as usual:
  • made backups
  • ran in shell:
Code:
unzip -d 388 chevereto_3.8.8.zip
cd 388/chevereto
cp -R .htaccess * /path/to/www/
  • ran mysql tables update
And then it stalls.
It spawns lot's of requests to database and whole site gets down.
SHOW PROCESSLIST shows queries to chv_users.
Rolled back to 3.8.3, but I want my updates.
I need help.

ps: I moved to a new server, so I can grant FTP/SSH access if needed.
 
Last edited:
First use a new edition, v3.8.8 is quite old.

Now, in huge websites or websites with too low resources left you should do a manual install:
  1. Download the package and upload it to your server
  2. Unzip via command line (you need to unzip /chevereto/* to your public folder
  3. Go to /install?debug
By doing that you will be prompted with all the queries that you need to run.
 
3.Go to /install?debug
Now I tried recent version 3.8.13.
Set to Maintenance mode.
Updated all the files. Went to /install?debug. Waited for 30s (script execution timeout). 504.
Now php-fpm does not accept any more connections so I get 502.
MySQL has one query in progress and 4 queued. And this doesn't seem to end.
SHOW PROCESSLIST:
Code:
88748527    root    localhost:52434    devchev_db    Query    0    init    SHOW PROCESSLIST    0.000
88748528    root    localhost:52435    NULL    Sleep    36        NULL    0.000
88748583    devchev    localhost    devchev_db    Query    3    Waiting for table level lock    UPDATE `chv_users` SET user_likes = user_likes - COALESCE((SELECT COUNT(*) FROM `chv_likes` WHERE li    0.000
88748589    devchev    localhost    devchev_db    Query    3    Waiting for table level lock    UPDATE `chv_users` SET user_likes = user_likes - COALESCE((SELECT COUNT(*) FROM `chv_likes` WHERE li    0.000
88748593    devchev    localhost    devchev_db    Query    3    Waiting for table level lock    UPDATE `chv_users` SET user_likes = user_likes - COALESCE((SELECT COUNT(*) FROM `chv_likes` WHERE li    0.000
88748594    devchev    localhost    devchev_db    Query    3    Sending data    UPDATE `chv_users` SET user_followers = user_followers - COALESCE((SELECT 1 FROM `chv_follows` WHERE    0.000
88748608    devchev    localhost    devchev_db    Query    3    Waiting for table level lock    UPDATE `chv_users` SET user_likes = user_likes - COALESCE((SELECT COUNT(*) FROM `chv_likes` WHERE li    0.000
Site is not operational.

Rolled files back to 3.8.3. Restarted MySQL. Site is operational again.
Healthy SHOW PROCESSLIST:
Code:
1513    root    localhost:52480    devchev_db    Query    0    init    SHOW PROCESSLIST    0.000
 
/install?debug doesn't run one single query and it doesn't touch any single file that could cause a hang.

If you go to /install?debug and you get a timeout is because your server is busy with something else, based in your log, is the previous update attempt. At this point you should stop all those queries, run mysql repair and then do all the /install?debug process again.

This happens because your MySQL server is just rubbish for the size of your website.
 
It seems that the server is busy deleting users. I run each update from working state with no background tasks. So it doesn't seem to be a previous update attempt hang.
Does the app makes any queries when in maintenance mode?
This happens because your MySQL server is just rubbish for the size of your website.
I have low traffic. Site runs smooth on 3.8.3.
 
Can you explain this line of changelog:
Chevereto v3.8.4
Released on October 19th 2016
  • Added automatic removal of not confirmed user accounts
It seems that my server is busy doing that.

3.8.3 is the last version that I'm able to run so logically something has changed in 3.8.4
 
Last edited:
It seems that the server is busy deleting users.
System deletes unconfirmed users, those that never clicked confirmation link sent via email. System deletes only 5 accounts per load so if for example your website has 300 concurrent users is issuing 1,500 operations per second.

PHP:
    public static function cleanup() {
        $db = DB::getInstance();
        $db->query('SELECT * FROM ' . DB::getTable('users') . ' WHERE user_status IN ("awaiting-confirmation", "awaiting-email") AND user_date_gmt <= DATE_SUB(UTC_TIMESTAMP(), INTERVAL 2 DAY) ORDER BY user_id DESC LIMIT 5'); // Only 5 entries per round, this is an expensive job
        $users = $db->fetchAll();
        foreach($users as $user) {
            $user = self::formatArray($user);
            self::delete($user);
        }
    }

Does the app makes any queries when in maintenance mode?
It attempts to do all the /install stuff on maintenance mode and reverts the status on completion.

I have low traffic. Site runs smooth on 3.8.3.
If you have low traffic then is hard to explain that the delete operation takes that much.

Best that you can do is go to app/loader.php and change this:
PHP:
// Handle invalid user accounts
if(method_exists('CHV\User','cleanup')) {
    try {
        User::cleanup();
    } catch(Exception $e) {}
}

To this:
PHP:
// Handle invalid user accounts
if(method_exists('CHV\User','cleanup') && iiset($_GET['cleanup'])) {
    try {
        User::cleanup();
    } catch(Exception $e) {}
}

And open your website /?cleanup so the process will be issued only when you add that get request.
 
Wow. I discovered that I have about 50 users awaiting confirmation with more than 0 uploads. A total of 490 images uploaded by these users.
For example:
http://keep4u.ru/leila
Seems like a legit content. Not spam.
This cleanup will delete all their content.
I'm sure they'll be mad at me.
Maybe they forgot to confirm, or a confirmation went to spam or a mailserver failed to send messages.

I'll try your solution tomorrow. I'll change that file before I'll replace old files with new ones so this line won't run anyway.

If you have low traffic then is hard to explain that the delete operation takes that much.
Search robots do a lots of requests. They are not counted as traffic.

It attemps to do all the /install stuff on maintenance mode and reverts the status on completion.
No. I mean when I set maintenance mode 1, there should be no DB queries other than settings or admin tasks. But even when in maintenance, app still performs cleanup. At least if I'm getting the maintenance mode right.
 
Wow. I discovered that I have about 50 users awaiting confirmation with more than 0 uploads. A total of 490 images uploaded by these users.

Yep, also have none-confirmed users with images. Solution is edit cleanup function app\lib\classes\class.user.php
Code:
$db->query('SELECT * FROM ' . DB::getTable('users') . ' WHERE user_status IN ("awaiting-confirmation", "awaiting-email") AND user_date_gmt <= DATE_SUB(UTC_TIMESTAMP(), INTERVAL 2 DAY) ORDER BY user_id DESC LIMIT 5'); // Only 5 entries per round, this is an expensive job

Replace with:
Code:
$db->query('SELECT * FROM ' . DB::getTable('users') . ' WHERE user_status IN ("awaiting-confirmation", "awaiting-email") AND user_image_count = 0 AND user_date_gmt <= DATE_SUB(UTC_TIMESTAMP(), INTERVAL 2 DAY) ORDER BY user_id DESC LIMIT 5'); // Only 5 entries per round, this is an expensive job

Now I tried recent version 3.8.13.
Set to Maintenance mode.
Updated all the files. Went to /install?debug. Waited for 30s (script execution timeout). 504.
Now php-fpm does not accept any more connections so I get 502.
MySQL has one query in progress and 4 queued. And this doesn't seem to end.
SHOW PROCESSLIST:
Site is not operational.

Probably this will help you. Remove stats query in app\install\installer.php line #1413
You really do not need it.
Code:
                        // Stats (since 3.7.0)
                        $install_sql .= strtr($stats_query, [
                                '%table_prefix%' => $table_prefix,
                                '%table_engine%' => $fulltext_engine
                            ]);
 
Someone who never confirmed the account will be mad because you delete that content?

Come on... That user never confirmed the account and the system deletes those who attempted to create an account and the link got expired.

User cleanup will remain the same, those who want to keep that content go ahead and edit the code to suit your your needs.
 
Probably this will help you. Remove stats query in app\install\installer.php line #1413
You really do not need it.
Problem is not the stats query, is not being even executed when updating from 3.8.8 to 3.8.13. By the way, since you are interested in this kind of stuff... Problem here is that the system should use a lock file (is a dirty cheap way in which the process "locks" successive operations and avoids this kind of stuff) 99% of the websites out there don't need this at all but at some point we will need it anyway so yeah, this was my fault for being lazy.

Cheers,
Rodolfo
 
Last edited:
Come on... That user never confirmed the account and the system deletes those who attempted to create an account and the link got expired.

Rodolfo - you know that 80% of your clients has no experience to manage there servers. They failed already to set the right permission or just too lazy to read the documentation.
So I would not confirm, that it is a "bad user" if he not confirms the account. Maybe he just not received the confirmation message in case of wrong configured mail systems. I think, most of the "admins" will use PHP mailer - where most of the mails going into Spam ...
 
I think, most of the "admins" will use PHP mailer - where most of the mails going into Spam ...

True, but in that case the system allows you to use your own SMTP delivery, is cheap (actually free in most tiers) and I don't see why some still use phpmail.

In any case it seems that I need to write more and more documentation about everything really, hope that some will read it.

;)
 
I used both Rodolfo's and Fallout's solutions. I changed the GET parameter to a code, so noone will hang my service just after reading this thread :cool: I found that it takes me about 58s to finish one cleanup. Not sure what to do with it.
And I think it's safe to delete users with no images. As for others, I'll send a mail to them to respond. If they won't respond, I'll ban them. At least if they'll appeal to support I'll be able to restore their content.

Update complete
I'm running 3.8.13
Thank you guys
 
Problem is not the stats query

Rodolfo, my fault, you right. PS: I never updated via install.php

I found that it takes me about 58s to finish one cleanup. Not sure what to do with it.

Look at query time after # some query need optimization, tested on table with 10k users

Code:
DEBUG ( G\DB::exec ) DELETE FROM `chv_users` WHERE `user_id`=:user_id # 0.001
DEBUG ( G\DB::exec ) DELETE FROM `chv_follows` WHERE `follow_user_id`=:follow_user_id OR `follow_followed_user_id`=:follow_followed_user_id # 0
DEBUG ( G\DB::exec ) DELETE FROM `chv_likes` WHERE `like_user_id`=:like_user_id # 0
DEBUG ( G\DB::exec ) DELETE FROM `chv_logins` WHERE `login_user_id`=:login_user_id # 0.001
DEBUG ( G\DB::exec ) DELETE FROM `chv_images` WHERE `image_user_id`=:image_user_id # 0
DEBUG ( G\DB::exec ) DELETE FROM `chv_albums` WHERE `album_user_id`=:album_user_id # 0
DEBUG ( G\DB::exec ) UPDATE `chv_users` SET user_following = user_following - COALESCE((SELECT 1 FROM `chv_follows` WHERE follow_followed_user_id = 10273 AND user_id = follow_user_id AND follow_user_id <> follow_followed_user_id GROUP BY follow_user_id),"0"); # 0.124
DEBUG ( G\DB::exec ) UPDATE `chv_users` SET user_followers = user_followers - COALESCE((SELECT 1 FROM `chv_follows` WHERE follow_user_id = 10273 AND user_id = follow_followed_user_id AND follow_user_id <> follow_followed_user_id GROUP BY follow_followed_user_id),"0"); # 0.118
DEBUG ( G\DB::exec ) UPDATE `chv_users` SET user_likes = user_likes - COALESCE((SELECT COUNT(*) FROM `chv_likes` WHERE like_user_id = 10273 AND user_id = like_content_user_id AND like_user_id <> like_content_user_id GROUP BY like_content_user_id),"0"); # 0.129
DEBUG ( G\DB::exec ) UPDATE `chv_stats` SET stat_users = GREATEST(stat_users - 1, 0) WHERE stat_type = "total"; UPDATE `chv_stats` SET stat_users = GREATEST(stat_users - 1, 0) WHERE stat_type = "date" AND stat_date_gmt = DATE("2017-04-04 19:04:39");UPDATE IGNORE `chv_stats` AS S INNER JOIN ( SELECT DATE(like_date_gmt) AS like_date_gmt, COUNT(*) AS cnt FROM `chv_likes` WHERE like_user_id = 10273 GROUP BY DATE(like_date_gmt) ) AS L ON S.stat_date_gmt = L.like_date_gmt SET S.stat_likes = GREATEST(S.stat_likes - COALESCE(L.cnt, "0"), 0) WHERE stat_type = "date"; UPDATE IGNORE `chv_stats` SET stat_likes = GREATEST(stat_likes - COALESCE((SELECT COUNT(*) FROM `chv_likes` WHERE like_user_id = 10273), "0"), 0) WHERE stat_type = "total"; UPDATE IGNORE `chv_stats` AS S INNER JOIN ( SELECT DATE(album_date_gmt) AS album_date_gmt, COUNT(*) AS cnt FROM `chv_albums` WHERE album_user_id = 10273 GROUP BY DATE(album_date_gmt) ) AS A ON S.stat_date_gmt = A.album_date_gmt SET S.stat_albums = GREATEST(S.stat_albums - COALESCE(A.cnt, "0"), 0) WHERE stat_type = "date"; UPDATE IGNORE `chv_stats` SET stat_albums = GREATEST(stat_albums - COALESCE((SELECT COUNT(*) FROM `chv_albums` WHERE album_user_id = 10273), "0"), 0) WHERE stat_type = "total"; # 0
DEBUG ( G\DB::exec ) UPDATE IGNORE `chv_users` AS U INNER JOIN ( SELECT notification_user_id, COUNT(*) AS cnt FROM `chv_notifications` WHERE notification_trigger_user_id = 10273 AND notification_is_read = 0 GROUP BY notification_user_id ) AS N ON U.user_id = N.notification_user_id SET U.user_notifications_unread = GREATEST(U.user_notifications_unread - COALESCE(N.cnt, "0"), 0); DELETE IGNORE `chv_notifications` FROM `chv_notifications` LEFT JOIN `chv_follows` ON notification_type_id = follow_id AND follow_user_id = 10273 LEFT JOIN `chv_likes` ON notification_type_id = like_id AND like_user_id = 10273 WHERE (notification_type = "follow" AND notification_type_id = follow_id) OR (notification_type = "like" AND notification_type_id = like_id); DELETE IGNORE FROM `chv_notifications` WHERE notification_user_id = 10273; # 0.001
DEBUG ( G\DB::exec ) SELECT image_id FROM chv_images WHERE image_user_id=:image_user_id # 0
DEBUG ( G\DB::exec ) SELECT * FROM chv_users WHERE user_status IN ("awaiting-confirmation", "awaiting-email") AND user_image_count = 0 AND user_date_gmt <= DATE_SUB(UTC_TIMESTAMP(), INTERVAL 2 DAY) ORDER BY user_id DESC LIMIT 5 # 0.007

My temprary fix for function delete($user), file app\lib\classes\class.user.php
PHP:
public static function delete($user) {
        try {
            if(!is_array($user)) {
                $user = self::getSingle($user, 'id', TRUE);
            }
            // Delete content user image folder
            $user_images_path = CHV_PATH_CONTENT_IMAGES_USERS . $user['id_encoded'];
            if(!@unlink($user_images_path)) {
                $files = glob($user_images_path.'/{,.}*', GLOB_BRACE);
                foreach($files as $file){
                    if(is_file($file)) {
                        @unlink($file);
                    }
                }
            }
            
            // Delete images from disk
            $db = DB::getInstance();
            $db->query('SELECT image_id FROM '.DB::getTable('images').' WHERE image_user_id=:image_user_id');
            $db->bind(':image_user_id', $user['id']);
            $user_images = $db->fetchAll();
            
            foreach($user_images as $user_image) {
                Image::delete($user_image['image_id']);
            }

            // Remove related notifications tied to this user (follows)
            Notification::delete([
                'table'        => 'users',
                'user_id'    => $user['id'],
            ]);
            
            // Track stats
            Stat::track([
                'action'    => 'delete',
                'table'        => 'users',
                'value'        => '-1',
                'user_id'    => $user['id'],
                'date_gmt'    => $user['date_gmt']
            ]);
            
            // Update affected user_likes count
            $sql = strtr('UPDATE `%table_users` SET user_likes = user_likes - COALESCE((SELECT COUNT(*) FROM `%table_likes` WHERE like_user_id = %user_id AND user_id = like_content_user_id AND like_user_id <> like_content_user_id GROUP BY like_content_user_id),"0") WHERE user_likes > 0;', [
                '%table_users'    => DB::getTable('users'),
                '%table_likes'    => DB::getTable('likes'),
                '%user_id'        => $user['id'],
            ]);
            DB::queryExec($sql);

            // Update affected user_liked count (users who liked content owner by this user)
            // --> Should happen in Image::delete()
            
            // Update affected user_followers count
            $sql = strtr('UPDATE `%table_users` SET user_followers = user_followers - COALESCE((SELECT 1 FROM `%table_follows` WHERE follow_user_id = %user_id AND user_id = follow_followed_user_id AND follow_user_id <> follow_followed_user_id GROUP BY follow_followed_user_id),"0") WHERE user_followers > 0;', [
                '%table_users'    => DB::getTable('users'),
                '%table_follows'=> DB::getTable('follows'),
                '%user_id'        => $user['id'],
            ]);
            DB::queryExec($sql);
            
            // Update affected user_following count
            $sql = strtr('UPDATE `%table_users` SET user_following = user_following - COALESCE((SELECT 1 FROM `%table_follows` WHERE follow_followed_user_id = %user_id AND user_id = follow_user_id AND follow_user_id <> follow_followed_user_id GROUP BY follow_user_id),"0") WHERE user_following > 0;', [
                '%table_users'    => DB::getTable('users'),
                '%table_follows'=> DB::getTable('follows'),
                '%user_id'        => $user['id'],
            ]);

            DB::queryExec($sql);
            
            DB::delete('albums', ['user_id' => $user['id']]); // Delete albums DB
            DB::delete('images', ['user_id' => $user['id']]); // Delete images DB
            DB::delete('logins', ['user_id' => $user['id']]); // Delete logins
            DB::delete('likes', ['user_id' => $user['id']]); // Delete user likes
            DB::delete('follows', ['user_id' => $user['id'], 'followed_user_id' => $user['id']], 'OR'); // Delete user's followers and follows
            DB::delete('users', ['id' => $user['id']]); // Delete user DB

        } catch(Exception $e) {
            throw new UserException($e->getMessage(), $e->getCode());
        }
    }
Great runtime now
Code:
DEBUG ( G\DB::exec ) UPDATE `chv_users` SET user_likes = user_likes - COALESCE((SELECT COUNT(*) FROM `chv_likes` WHERE like_user_id = 9965 AND user_id = like_content_user_id AND like_user_id <> like_content_user_id GROUP BY like_content_user_id),"0") WHERE user_likes > 0; # 0.001
DEBUG ( G\DB::exec ) UPDATE `chv_users` SET user_followers = user_followers - COALESCE((SELECT 1 FROM `chv_follows` WHERE follow_user_id = 9965 AND user_id = follow_followed_user_id AND follow_user_id <> follow_followed_user_id GROUP BY follow_followed_user_id),"0") WHERE user_followers > 0; # 0.001
DEBUG ( G\DB::exec ) UPDATE `chv_users` SET user_following = user_following - COALESCE((SELECT 1 FROM `chv_follows` WHERE follow_followed_user_id = 9965 AND user_id = follow_user_id AND follow_user_id <> follow_followed_user_id GROUP BY follow_user_id),"0") WHERE user_following > 0; # 0.001

Also before run this heavy query we can check if user have any user_liked/user_following/user_followers values
 
Problem aren't the queries but the way on which the system perform those calls without any cap meaning that it could hang any server because of that. Queries itself are not expensive (0.124 s) and those are needed in this DB structure but is different to execute 10 times this (1.24s) vs 100 times (12.4s) vs 1,000 (124s) and problem here is that the system is not dealing with concurrency.

System has 3 bottlenecks related to concurrency at this time:
  • User cleanup (this issue is all about that)
  • Delete queue handler (external storage)
  • Updates check
The fix is quite easy and elegant, I will include it in next release.

Cheers,
Rodolfo.
 
Your temporary fix breaks likes count.

How is break? It just skip rows with zero values. Zero likes remain zero likes always.

PS: for cleunup function we can skip queries, as non-confirmed users always have zero likes, following etc.

Problem aren't the queries but the way on which the system perform those calls without any cap meaning that it could hang any server because of that. Queries itself are not expensive (0.124 s) and those are needed in this DB structure but is different to execute 10 times this (1.24s) vs 100 times (12.4s) vs 1,000 (124s) and problem here is that the system is not dealing with concurrency.

Not. For gobzer site the query is main problem. For table with 992,654 Users it check every single row, I bet for him 1 query run cost 10-20 seconds!

System has 3 bottlenecks related to concurrency at this time:
  • User cleanup (this issue is all about that)
  • Delete queue handler (external storage)
  • Updates check

Agree, why I select separated cron file for these task.
 
Last edited:
How is break? It just skip row with zero values. Zero likes remain zero likes always.
Users could like content and could get liked content, in any case you need to run a query to fix those counts.

Not. For gobzer site the query is main problem. For table with 992,654 Users it update every table rows, I bet for him 1 query run cost 10-20 seconds!
Is a concurrency issue! I've explained how and when it happens and I even mentioned how I will fix it.

Topic closed.
 
Status
Not open for further replies.
Back
Top