Image deletion queue is not being processed on b2 external storage

[lovedigit]

▶ Reproduction steps

1. Make sure you have external storage set to backblaze b2
2. Upload an image
3. delete an image

😢 Unexpected result

Deletion queue is not being processed, even though cron job is setup correctly and was working on v3.18.3. About 60K images are waiting to be processed:



📃 Error log message

I don't have any log for this. Please point me how to get it.
As far as I know it is not an sftp request, and chevereto is using b2 api?

Edit: v3.19.beta.2 allows better visualization on errors. Thanks to rodolfo.

[CODE title="error log"]Aw, snap! Internal Server Error [debug @ error_log] - https://v3-docs.chevereto.com/setup/debug.html

Fatal error [0]:
Triggered in /var/www/lensdump/html/app/vendor/esac/backblaze-b2/src/Client.php:424

Stack trace:
#0 /var/www/lensdump/html/app/vendor/esac/backblaze-b2/src/Client.php(452): esac\B2\Client->getFile()
#1 /var/www/lensdump/html/app/lib/classes/class.queue.php(161): esac\B2\Client->deleteFile()
#2 /var/www/lensdump/html/app/cron.php(39): CHV\Queue:😛rocess()
#3 /var/www/lensdump/html/app/cron.php(26): storageDelete()
#4 /var/www/lensdump/html/app/loader.php(296): require_once('/var/www/lensdump/html/app/cron.php')
#5 /var/www/lensdump/html/cron.php(23): include_once('/var/www/lensdump/html/app/loader.php')[/CODE]

 

[JakeSully]

▶ Reproduction steps

1. Make sure you have external storage set to backblaze b2
2. Upload an image
3. delete an image

😢 Unexpected result

Deletion queue is not being processed, even though cron job is setup correctly and was working on v3.18.3. About 60K images are waiting to be processed:

View attachment 3753

📃 Error log message

I don't have any log for this. Please point me how to get it.
As far as I know it is not an sftp request, and chevereto is using b2 api?

another user had same thing, but they didn't have cron job located on index of dashboard setted up. make sure that cron job is setup for it to delete pending images.

If you don't it won't delete them!

Also check if cronjob file is giving out any errors.

 

[lovedigit]

Cron job is setup correctly, and crons are being processed every minute.
This was logged a minute ago:



I checked cron log. No error reported there.

 

[imghut]

You could try Lifecycle Settings > Keep only the last version of the file, just guessing here 😉

 

[lovedigit]

That is already set at b2 bucket @HenrysCat

 

[Rodolfo]

No error reported there.

>/dev/null 2>&1 nullifies the output of the command. If you nullify the command the log will be only at your error log. You can learn more about it at explainshell.

 

[lovedigit]

>/dev/null 2>&1 nullifies the output of the command. If you nullify the command the log will be only at your error log. You can learn more about it at explainshell.

Yeah, I ran the command manually with same php user that is bind with chevereto installation. It doesn't show any error.
And no image are deleted even when I run the cron manually.



No error in cron log, when ran through cronjob without output redirected to null.



Edit: And here is the cronjob output:

 

[JakeSully]

I deleted a image in 3.19.0 beta 1.

It took few mins before image at backblaze b2 was deleted. I keept force refreshing it so it reads new cache and then after maybe 3 mins or 2 it said 404 not found.

So it seems to work fine for me.

 

[lovedigit]

It was working in 3.18.3.
I did not change anything.
As you can see, cron jobs are working, manually triggering the cron works.
What could go wrong between updates?

Can you please check your database chv_queues table?
Also, if you can check with large number of images, that would be great. I have currently 59K pending after update, and not a single image is deleted.

Here is the cronjob output

 

[hyypia21]

It was working in 3.18.3.
I did not change anything.
As you can see, cron jobs are working, manually triggering the cron works.
What could go wrong between updates?

Can you please check your database chv_queues table?
Also, if you can check with large number of images, that would be great. I have currently 59K pending after update, and not a single image is deleted.

Here is the cronjob output

View attachment 3763

I have met the same problem. I have about 4K pendings in chv_queues table

 

[lovedigit]

I have met the same problem. I have about 4K pendings in chv_queues table

You've cron job setup correctly? And are you on 3.19.beta1?
Just want to make sure we both have same issue. And everything is setup correctly.

 

[JakeSully]

If more people have this problem, maybe it is a issue when it comes to when there is a specific amount of images that needs to be deleted, that is when it could be where it won't delete as it should?

Since i so far only deleted like 1-5 images and all of them where properly deleted at Backblaze b2 without a issue.

 

[lovedigit]

If more people have this problem, maybe it is a issue when it comes to when there is a specific amount of images that needs to be deleted, that is when it could be where it won't delete as it should?

Since i so far only deleted like 1-5 images and all of them where properly deleted at Backblaze b2 without a issue.

Hmm, I don't know. If it is specific to my setup, I am unsure where the problem might be.
My queue is only growing.

 

[JakeSully]

Hmm, I don't know. If it is specific to my setup, I am unsure where the problem might be.
My queue is only growing.

View attachment 3769

it might be your setup, since i just ran a deletion of 279 images all where put int queue table and shortly it was 101 images listed and now it's 0 queued, i also checked images on backblaze b2 and couldn't find matching files, so i can confirm it seems to work fine for me.

 

[lovedigit]

After updating to v3.19.beta.2 got an stack trace for error. Thanks to rodolfo's to help visualize the error.


[CODE title="error log"]Aw, snap! Internal Server Error [debug @ error_log] - https://v3-docs.chevereto.com/setup/debug.html

Fatal error [0]:
Triggered in /var/www/lensdump/html/app/vendor/esac/backblaze-b2/src/Client.php:424

Stack trace:
#0 /var/www/lensdump/html/app/vendor/esac/backblaze-b2/src/Client.php(452): esac\B2\Client->getFile()
#1 /var/www/lensdump/html/app/lib/classes/class.queue.php(161): esac\B2\Client->deleteFile()
#2 /var/www/lensdump/html/app/cron.php(39): CHV\Queue:😛rocess()
#3 /var/www/lensdump/html/app/cron.php(26): storageDelete()
#4 /var/www/lensdump/html/app/loader.php(296): require_once('/var/www/lensdump/html/app/cron.php')
#5 /var/www/lensdump/html/cron.php(23): include_once('/var/www/lensdump/html/app/loader.php')
[/CODE]

 

[Rodolfo]

Check the trace and you will note that the system query the B2 API for the file id. To do that, they require the bucket and the file key (path). With that info it returns a list of files and from there it iterates that list to find the matching file.

The trace is saying that such get file ID process is failing. Why failing for some cases only? I believe that it could be bucket size dependant, like a huge bucket could be enough to cause this kind of failure. This kind of stuff requires debugging for that dependency, and to detect the conditions needed to replicate this.

Either B2 API is not working as expected or the dependency is wrong. Best you can do is to show me more cases, if we are lucky we could narrow the issue and either fix the package or request B2 to fix the service.