External storage images not getting deleted

mkerala

Network license
Internals
V3 Beta tester
License owner
Big Chevereto
Recently noticed that direct links of images are still accessible even after deleting the images. Checked the storage location and files are still there despite image being deleted from the site and database. Tested by uploading a new image and deleting it. The file still remains in the external storage.

To confirm external storage account has the correct permission, logged into storage as the same user and was able to delete the file manually via SFTP. Tested on demo, but could not reproduce as demo is not using external storage.

▶🚶‍Reproduction steps
  1. Login as a user to a Chevereto site with external storage.
  2. Upload a any image file.
  3. Copy the direct image link.
  4. Delete the image and try accessing the image via direct link.
😢Unexpected result

The image is still active even after being deleted from the site and database.

📃Error log message

Enabled SFTP logs on the server and monitored all activity during upload and delete. Logs show activity for file being uploaded. But no activity when file is being deleted. Not even a connection attempt.
 

Rodolfo

Don Chevere
Chevereto Staff
Administrator
License owner
If the image is gone from the db is because it got an OK from the the external removal step. Does it happens with all storages? If not, then the issue is your external storage server.
 

mkerala

Network license
Internals
V3 Beta tester
License owner
Big Chevereto
I got two sftp file servers which has not had any changes being made for the last one year. Everything was working perfectly fine before I updated to 3.17.1 two days ago. I logged in via Filezilla as the same user account Chevereto uses and deleted the files. So there is no server side errors or permission issues.

Below is the full SFTP log. As you can see logs is showing all activity when the file was added. But nothing, not even a connection attempt when the image was deleted.

Code:
Dec 30 13:28:15 storage sftp-server[2987675]: session opened for local user chevftp from [94.130.xxx.xx]
Dec 30 13:28:15 storage sftp-server[2987675]: received client version 3
Dec 30 13:28:15 storage sftp-server[2987675]: realpath "."
Dec 30 13:28:15 storage sftp-server[2987675]: opendir "/var/www/html/s2/images"
Dec 30 13:28:15 storage sftp-server[2987675]: closedir "/var/www/html/s2/images"
Dec 30 13:28:15 storage sftp-server[2987675]: open "/var/www/html/s2/images/DyWSDqsdQSDqsdqsdHG.png" flags WRITE,CREATE,TRUNCATE mode 0666
Dec 30 13:28:15 storage sftp-server[2987675]: close "/var/www/html/s2/images/DyWSDqsdQSDqsdqsdHG.png" bytes read 0 written 175389
Dec 30 13:28:15 storage sftp-server[2987675]: open "/var/www/html/s2/images/DyWSDqsdQSDqsdqsdHG.md.png" flags WRITE,CREATE,TRUNCATE mode 0666
Dec 30 13:28:15 storage sftp-server[2987675]: close "/var/www/html/s2/images/DyWSDqsdQSDqsdqsdHG.md.png" bytes read 0 written 457652
Dec 30 13:28:15 storage sftp-server[2987675]: open "/var/www/html/s2/images/DyWSDqsdQSDqsdqsdHG.th.png" flags WRITE,CREATE,TRUNCATE mode 0666
Dec 30 13:28:15 storage sftp-server[2987675]: close "/var/www/html/s2/images/DyWSDqsdQSDqsdqsdHG.th.png" bytes read 0 written 34684
Dec 30 13:28:15 storage sftp-server[2987675]: session closed for local user chevftp from [94.130.xxx.xx]
Dec 30 13:28:46 storage sftp-server[2987588]: realpath "/var/log/sftp.log"
Dec 30 13:28:46 storage sftp-server[2987588]: stat name "/var/log/sftp.log"
Dec 30 13:28:47 storage sftp-server[2987588]: open "/var/log/sftp.log" flags READ mode 0666
 

mkerala

Network license
Internals
V3 Beta tester
License owner
Big Chevereto
I got another Chevereto installation running v3.15.2. Tested the same external storage with the same settings(same URL too) and images are getting deleted.
 

Rodolfo

Don Chevere
Chevereto Staff
Administrator
License owner
Interesting, let me know if you find anything else.
 

mkerala

Network license
Internals
V3 Beta tester
License owner
Big Chevereto
I was able to test on a Chevereto 3.17.0 install of a friend. Added the same external storage and was able to replicate the issue.

Also found auto expire image is not working as well.
  • Affects only SFTP external storage.
  • Affects version 3.17 or higher
  • Will break the auto expire
I will wait for others using external storage to test this.
 

HenrysCat

Core license
License owner
I tested this my own site v3.17.1 as I also use SFTP external storage, and I found once the image is deleted it takes 1 minute 15 seconds to be removed from the SFTP server, I have tested this 3 times now, hope this helps.
 

mkerala

Network license
Internals
V3 Beta tester
License owner
Big Chevereto
I tested this my own site v3.17.1 as I also use SFTP external storage, and I found once the image is deleted it takes 1 minute 15 seconds to be removed from the SFTP server, I have tested this 3 times now, hope this helps.
Thanks for testing. In my case the file stays there forever.

Did some further testing by adding a Hostgator SFTP as external storage. Same thing, file stays there for ever.

Another issue I noticed is the stats are not getting updated as well. If I click on search content, there is no image, yet shows 3.14 MB used.

1609395788278.png
 

HenrysCat

Core license
License owner
The Active box should be ticked right? also what are the permissions and owner of the storage folder?

Permissions should be 755 and owner/group should the user with the highest permissions in /etc/passwd
 

mkerala

Network license
Internals
V3 Beta tester
License owner
Big Chevereto
The Active box should be ticked right? also what are the permissions and owner of the storage folder?

Permissions should be 755 and owner/group should the user with the highest permissions in /etc/passwd
Yes for both.
 

HenrysCat

Core license
License owner
Try adding the storage folder in a directory that is owned the SFTP user and not the default /var/www/html/
 
Last edited:

HenrysCat

Core license
License owner
It's not, your logs posted above show /var/www/html/s2/images/ where /var/www/html/ is the default web folder, a user folder will be something along the lines of /home/user/web/public_html or similar.
 

mkerala

Network license
Internals
V3 Beta tester
License owner
Big Chevereto
It's not, your logs posted above show /var/www/html/s2/images/ where /var/www/html/ is the default web folder, a user folder will be something along the lines of /home/user/web/public_html or similar.
I can't use home folder as it is not mapped to Nginx and image won't load. /var/www/html/s2/images/ this folder is owned by the sftp user.

I even tried with Hostgator sftp and got the same results. So I don't think its an issue with the server. Besides the same folder and user works fine with my 3.15 Chevereto install.
 

HenrysCat

Core license
License owner
I can't use home folder as it is not mapped to Nginx and image won't load. /var/www/html/s2/images/ this folder is owned by the sftp user.
Use a Symlink and don't forget to set disable_symlinks off; in nginx.conf

Do you use a control panel?
 

evil

Core license
License owner
Hello,

I think you have the same problem as me, check your DB, go to "chv_queues" are cron jobs in "pending" status?
My images are removed from website "system" but not physically from external storage.

 

Rodolfo

Don Chevere
Chevereto Staff
Administrator
License owner
Hello,

I think you have the same problem as me, check your DB, go to "chv_queues" are cron jobs in "pending" status?
My images are removed from website "system" but not physically from external storage.

He already mentioned that the database is being updated, is just the actual files the problem. Unless he did a wrong check and the queues contain the failed attempts.
 

Rodolfo

Don Chevere
Chevereto Staff
Administrator
License owner
I just tested Amazon S3 cron removal and it works. Like I said before, try using other external storage (api) to determine where the issue is. If you keep testing SFTP you don't bring anything different to the process and you won't be able to tell any difference.

Perhaps the issue is the implentation, certificates, the cron user, whatever. Bottom line is do some change and report back, otherwise we will here forever running in circles.
 

mkerala

Network license
Internals
V3 Beta tester
License owner
Big Chevereto
Hello,

I think you have the same problem as me, check your DB, go to "chv_queues" are cron jobs in "pending" status?
My images are removed from website "system" but not physically from external storage.

Thanks. This pointed me in the right direction. Indeed the jobs were pending for storage delete.

Chevreto 3.16 introduced a new requirement for a cron job https://v3-docs.chevereto.com/setup/requirements.html#cron. Even though the image get removed from DB, storage delete are queued for Cron job. This explains why it takes up to 2 mins for the actual file to be deleted.

Added cron job as root and queue started clearing. Took a while as there was one week of pending delete jobs.
 
Last edited:
Top