New uploads are a mess after removing locks

[Stam]

Website URL
<private>

Chevereto version
3.10.5

Description of the issue
Initial issue:
https://chevereto.com/community/threads/file-naming-method-broken.9349/

After removing the files inside that folder as instructed, the URL and filename are now working. But it seems to be generating the same IDs for new uploads that existed before and it's creating a mess.

I have cloudflare cache enabled, and even though I purged everything in the cache, the problem remains because it needs to be purged every time a new upload is made.

People are uploading a new image and see some other image instead... let alone the medium sized preview pic that might not change because new image are too small to have such size generated.

Example:
https://fmshots.com/images/DBTli.png
https://fmshots.com/images/DBTli.md.png

What can I do to prevent this?

 

[Rodolfo]

Send me a server access, I need to see on my own.

 

[Rodolfo]

I still don't get server access.

 

[Stam]

Sorry, I've been really busy with real life commitments and couldn't deal with this issue earlier.

What details do you need?

I'm using this as a cdn to a much larger community and this has affected a lot of articles and forum posts that it's impossible to even track down and try to fix manually.

 

[Rodolfo]

I need server access, send the details to inbox@chevereto.com

 

[Stam]

I've just emailed you the details.

 

[Rodolfo]

Got it. I will check it on Monday.

 

[Rodolfo]

I've found two issues on your website.

The first issue is an ID conflict that happens very rarely and what it causes is that the ID doesn't match with the file name. From the perspective of the code, the most obvious reason for the ID issues is because of collisions on the lock system. The system locks the reserved ID, but it doesn't lock the process that reserves the ID itself:



It could be possible that N upload process happens almost at the exact same time and N images end with the same reserved ID but the insert command will ignore that and it will just add +1 to the ID. Also, PHP could be caching the $lock->check() method so it returns a false positive and therefore the ID reservation system crashes.

I've already forced a clearstatcache and I'm trying to get the error again without any luck (maybe it got fixed due to this subtle change).

This ID lock issue has nothing to do with MD/TH images being different from the original image and I can't find any logical explanation on why an upload will differ only on its MD or TH image because both sized images are generated almost at the same time and using the same file argument. For TH image it reads the source image and generates a thumbnail image, then for the MD image it reads the same source image and generates the medium sized image. It is unknown to me why PHP is reading wrong file contents for just one of these images.

The only explanation for what I described above is that PHP is not getting the file contents that it should get but I'm unable to tell you why that happens.

Check if this keeps happening, let me know when you find something odd.

 

[Stam]

I have cloudflare enabled. Want me to purge everything again?

After removing the files inside the lock folder, the problem started. It seemed like the same random filenames would be used and replace older pictures, probably because their ID was "unlocked" after removing those files.

If the newer images were not wide enough (in pixels; I believe I've set the minimum width to be 650 pixels for that) to generate the MD thumbnail, the older picture's MD thumbnail remained.

I'm guessing those pictures that have been replaced already, there's nothing we can do to undo/fix that. And hopefully the issue won't persist anymore after your subtle change.

 

[Rodolfo]

I have cloudflare enabled. Want me to purge everything again?

No, it is not a cache or a malfunction at that level. Is inside the machine and the thing is that it seems that the disk is not responding correctly.

After removing the files inside the lock folder, the problem started. It seemed like the same random filenames would be used and replace older pictures, probably because their ID was "unlocked" after removing those files.

That only explains when /<id> and /<id>.jpg aren't the same. It doesn't explain why <id>.th.jpg and <id>.md.jpg are different. To be honest, I don't know what causes that because it is almost impossible to replicate it. I spent several hours in your website and it never happened. It could be a slow disk, could be some sort of background process that messes with the disk, could be failing disk cache, php cache and a large etc.

What I did is that I forced some cache clean for the lock system so that should fix the lock system in your machine but that doesn't explain how two files generated from the same source ends up with different content.

This has been reported by other user and I wasn't able to replicate the bug either and my conclusion was that the filesystem was causing issues because the thing generates these files at the same time so it is impossible to get these results (different TH/MD) unless something isn't working properly in the server.

There are two issues here. That's what I'm trying to explain to you, ID locks control the file name but different th/md file content is something unknown for me and is not related to ID locks.

 

[Stam]

I believe different th/md is not the issue. The issue was newer uploads replacing older ones due to ID conflict.

I explained my reasoning for having different th/md in case of such conflict. If the newer picture wasn't wide enough to generate md thumbnail, it wouldn't remove the md that already existed (and therefore we'd see a different preview on list pages as they show md if it exists). Full size and th thumbnail are made for all pictures, so these would always change.

 

[Rodolfo]

Are you overriding IDs? That's explains everything.

I didn't code any failsafe operation for such cases because you should not be doing that. In order to save i/o systems do cache files (server, php, etc.) so you are doing something that you shouldn't.

When you have old files there, php won't be able to read the actual new file if the file is cached by the machine (which happens all the time) and there's no cache flush for that process. I added one now, but the system is designed to don't alter files that already exist.

I do understand everything now and that explains why it was so impossible to replicate this.

 

[Stam]

Erm, I mean I didn't do anything besides migrating the server. Didn't touch any settings. Just tried to explain what I figured out it's happening after removing the lock files.

 

[Stam]

In other words, I'm not sure exactly what it is I'm doing that I shouldn't be doing, nor how to fix it.

How can we ensure that IDs that already exist (they're all saved in 1 folder), won't be used again for future uploads?

 

[Rodolfo]

When you use ID filenames the system forces these to override any existing file with that name and from there, a re-cache instruction is sent to force the filesystem to acknowledge the new file contents but that won't work if your server keeps delivering the old file. And I meant internally, not externally (URL).

My guess is that this process doesn't happen fast enough for some images and your server delivers the old file because Chevereto doesn't wait, it assumes that you don't have any conflict there and that the files aren't cached.

There's no way to guarantee that this won't keep happening if you have images that shouldn't be at /images in the first place and not because these old images are there, is because your filesystem has a lag that messes this process.

 

[Stam]

I don't want older files to be replaced though, at all.
Is there a way to generate lock files for currently existing pictures inside the "images" folder?
Or my only option right now is to change file naming method from ID to something else?

 

[Rodolfo]

The lock system is used to reserve an ID before the actual database row insertion so the system generates a lock everytime you want to upload something and you use that filenaming method. The system scans the locks directory and always use a free ID.

Since the system needs to scan that directory, image ID locks are removed every 5 minutes so the upload process won't spend extra time reading a crowded directory. You can't use ID locks to prevent the system from using any given ID.

The problem here is that you mixed content and the thing that you can do is to change to another filenaming method or get rid of the files that shouldn't be there anyway.

Another alternative is that you alter the AUTO_INCREMENT value of chv_images table so you set a number that won't conflict with your old content. You need to know the gap (last ID of your old image) and add that to the current AUTO_INCREMENT value. By doing that the system will just skip these IDs and it won't conflict because there's no collision.

 

[Stam]

I still don't understand how I mixed content and what files shouldn't be there
Like I said, I merely migrated to a new server.

 

[Stam]

To be sure, I've changed the file naming method to "original" now.

I think there needs to be some operation/option added to Chevereto when you migrate server, so that this problem doesn't occur. I mean I haven't really done anything I shouldn't have, I just migrated to new server. Initially, the locks would make IDs of URL and filename to be different, and once we removed the lock files, it would start replacing older pictures because of reusing same IDs.

Thanks for taking the time to help me out, it's much appreciated.
And keep up the great work you do here, this is a brilliant product!

 

[Rodolfo]

We are not getting anywhere here. I need complete and accurate information or we will be hours here in circles and we won't get anywhere.

Can you confirm the following?

1. The contents of /image reflect exactly the contents of chv_images
2. You never added extra images to /images without passing them through Chevereto
3. You never used that folder to store images from anywhere else (even another Chevereto installation)