Images missing, albums disappeared, switched images since hitting 1 million+

[ashkir]

Website URL
https://nickpic.host

Chevereto version
3.10.18

Description of the issue
Hello. Ever since we hit 1 million+ we've been getting a plethora of switched images reported. We're having users' entire albums disappearing and more.

For example: https://nickpic.host/motherofgremlin this user had 435 images yesterday, all of them disappeared when she logged in this morning. Some of her missing images are:
https://nickpic.host/image/v2eVvx
https://nickpic.host/image/vMOes1
https://nickpic.host/image/vr8MkW

I'm having several users try to sort images into "albums" and then come back 3-4 hours later, the album is gone, and no longer exists and images are no longer assigned.

I'm having images with switched URLs, and have several reports in the last 24 hours now:
https://nickpic.host/image/v20JJY should've been an image called "true colors". Funny thing is the "album cover" uses the correct thumbnail:
https://drh2acu5z204m.cloudfront.net/items/060V0x2w33073J3E3G2E/Image 2018-07-13 at 10.12.10 AM.png It is also now listed under a different user entirely.

We recently hit the 1 million image mark. I don't know if that hurts or helps. But, this is throwing us off and we have absolutely no idea what to do. Is our database being overwhelmed, did we run out of unique ID#s?

As a side note, another site recently hit 1 million images too, and this same issue is happening to them. But, they haven't created a report yet. I just uploaded an image to them and it happens to them too. I'm PMing the owner so they may make a report too.


Something changed since the latest updates. These reports started occuring with greater frequency since .15 I don't know if it's to do with the code, or something deeper.

Where would I even begin troubleshooting an issue like this? It is to the point where I shut down all uploads because I cannot even stabalize it. The SQL server is fine, we use MariaDB. The file servers are fine and don't have hiccups. What do I do?

Edit: I did some slight googling, apparently MariaDB may have duplicate issues as it allows a row to be edited at any time/added in conjunction with other rows unless the software uses SQL coding that says DISTINCT or UNIQUE. Does Chevereto ensure the row is unique?

Edit 2: I had someone message me with this: https://nickpic.host/image/vI95ef This person uploaded this a few months ago, when they went back to it today, completely new image.

 

[mkerala]

I got 2+ million images and never had this issue and I am also running MariaDB. However, I upgraded to 3.10.18 today.

How do I replicate this issue?

 

[Rodolfo]

At script side this should fail whatever you have 100 images or 1,000,000. The script for handling these is exactly the same so if you manage to replicate this in the demo then we can talk about a bug in the system.

If you can't replicate it in the demo or anywhere else, then the issue is most likely your mysql server which is either not configured properly or it is just too busy and it corrupts data.

 

[Rodolfo]

For example: https://nickpic.host/motherofgremlin this user had 435 images yesterday, all of them disappeared when she logged in this morning.

The deletions have been made directly from the system. You can check it by running this query:

SELECT * FROM `chv_deletions` WHERE `deleted_content_user_id` = 1

Change "1" to reflect the actual user ID you are looking for.

My guess is that A) The user had enabled automatic deletions. B) Admin forced automatic deletions. C) There's some bug in the system that gets triggered upon certain circumstances. D) You or someone else added code that performs unwanted actions.

I've created a test album and from there I will check how it goes: https://nickpic.host/album/r6yi8

We recently hit the 1 million image mark. I don't know if that hurts or helps. But, this is throwing us off and we have absolutely no idea what to do. Is our database being overwhelmed, did we run out of unique ID#s?

The ids are capped to bigtint(32) which is 2,147,483,647 so I don't think you run out of unique ID#s, anyone will run out disk space way before run out of IDs. In any case, I will build a test case to detect possible collisions in the encodeID functionality.