Bulk Importer Via Cron?

[SaviorConnect]

🎯Description of the issue

Can you run the bulk importer via a cron? I have some very large import jobs of 30K+ images each. The total number of images will be over 5.5 million.

Additionally, if the import gets interrupted is any data lost or can it be safely resumed?

▶🚶‍Reproduction steps

1. N/A

😢Unexpected result

• N/A

📃Error log message

• N/A

 

[Rodolfo]

Can you run the bulk importer via a cron?

No, Chevereto V3 doesn't use cronjobs at all. The importer needs a web browser to work.

Additionally, if the import gets interrupted is any data lost or can it be safely resumed?

There's a "pause" action for that matter, so yes, you can resume the process.

 

[SaviorConnect]

Rodolfo,

Thank you for the reply.

The import process stops after about 20 images or so each time. If there is no option to auto-resume the upload...how do I proceed? The PHP max execution time is set to 90000. I have attached the log file. Chevereto is stopping the import for some reason.

I have millions of images to upload. One of the reasons I chose this software was for its ability to handle bulk imports. Even after this initial upload I have tens of millions of more images scheduled over the coming year.

Since I can't reply to my other ticket I will reply here. Can I at least process multiple import job ID's simultaneously?

Please advise as soon as possible. This project is time sensitive. I appreciate your understanding.

Blessings,
Steven Dunn

 

[Rodolfo]

A high execution time is not required for the importer. Usually PHP shouldn't be keep running for more that 30 seconds and when you increase that time you open a lot of other side issues (memory management, garbage collector, session timeout, etc).

The system detects the max execution time available and from there it attempts to use all the time possible, but it was never made/tested for 90,000 seconds, it was made for 30 seconds. If you put all that time then is likely that the session will be gone once the process attempts to process another batch. You could simply refresh the page and hit resume.

The bulk importer operates in batches and it was designed to handle unlimited number of images. It uses JavaScript to keep the process going (batches) so you don't need to put a huge time limit, just keep the browser tab open.

Additionally, it uses the database to store the progress so you can resume the process even in failure.

 

[SaviorConnect]

Rodolfo,

Thank you for the reply.

I don't mind it running in 30 second batches. However, there is no indication that the tab has to be left opened on the same page. You may want to make a mention in the instructions, on the page, or even add a "processing" gif while the AJAX is running. It may prevent similar questions in the future from others.

Where is the PHP file that manages the Bulk Importer?

Additionally, can I run multiple import jobs simultaneously? If I split my imports into separate jobs and ran them simultaneously I could complete the process much faster.

I appreciate the continued assistance and look forward to your reply.

Blessings,
Steven Dunn

 

[Rodolfo]

Where is the PHP file that manages the Bulk Importer?

The bulk importer works using several files, you should start with app/lib/classes/class.import.php

can I run multiple import jobs simultaneously? If I split my imports into separate jobs and ran them simultaneously I could complete the process much faster.

I've made some tests with additional threads and the outcome is not any better. In demo (DO cheap VPS):

1X 104 images 146s (0.7 img/s)
4X 136 images 164s (0.8 img/s)

(random high resolution images, between 5MB - 12MB taken from unsplash)

This is because the process is capped by the I/O, it should be faster with better hardware but in the end it needs to be re-made with a different logic to make it really faster. A logic involving service workers and less weight, getting rid of the web ui should also help to don't waste time in the HTTP request handling which adds a couple seconds between batches.