• Welcome to the Chevereto user community!

    Here users from all over the world gather around to learn the latest about Chevereto and contribute with ideas to improve the software.

    Please keep in mind:

    • 😌 This community is user driven. Be polite with other users.
    • 👉 Is required to purchase a Chevereto license to participate in this community (doesn't apply to Pre-sales).
    • 💸 Purchase a Pro Subscription to get access to active software support and faster ticket response times.
  • Chevereto Support CLST

    Support response

    Support checklist

    • ⚠️ Got a Something went wrong message? Read this guide and provide the actual error. Do not skip this.
    • ✅ Confirm that the server meets the System Requirements
    • 🔥 Check for any available Hotfix - your issue could be already reported/fixed
    • 📚 Read documentation - It will be required to Debug and understand Errors for a faster support response

Bulk Import - Missing Images

S

SaviorConnect

Chevereto Member
🎯Description of the issue

Bulk importer did not import all images. There are 31102 images in the directory but only 30997 were imported according to the stats. Please note however that all of the albums were imported correctly.

Note 2 - The bulk importer did "erase" all of the images int he bulk import directory. However, there are 5 images missing from the site.

Note 3 - I can provide you a copy of the file directory I uploaded if you need it for testing.

I would like to utilize Chevereto but I need all of my images to import reliably. I will appreciate your assistance in resolving this issue.

▶🚶‍Reproduction steps
  1. Process Bulk Import
  2. Missing images
😢Unexpected result

Not all images being imported from bulk importer.

📃Error log message

Log file - https://www.dropbox.com/transfer/AAAAAEqQ5ah64SBRjCLw16m7A_cEPejOLriTdJv68DzXGZEcML3Bfug
 
Sort by date Sort by votes
Any errors? The system has the same limits of normal upload, chances are that the skipped files are either duplicated, doesn't meet the upload parameters, or the images are corrupted.

You have to feed me the missing entries.

I feel that the bulk importer won't meet your expectations so I offer you a refund, I don't want you to feel cheated by the documentation / features.

Cheers.
 
Upvote 0 Downvote
Rodolfo,

Thank you for the reply.

You are correct, there was an issue with a few of the images. This issue was my fault.

I don't want to give up yet. This is an impressive piece of software that, presuming I get past these first few hurdles, will work great for my needs.

Additionally, based on the current traffic of my other online resources that I am migrating to Chevereto, I should be bringing it at least a million impressions a month (it's not a huge amount but it is growing aggressively). I also suspect I will probably be the largest Chevereto site very soon in sheer number of images. My point being is that I don't intend to waste your time and it should drive additional business your way.

I sincerely appreciate your assistance overcoming these initial hurtles and hope you will be willing to continue to support me. However, I understand if you are not interest and I won't take it personally.

Blessings,
Steven Dunn
 
Upvote 0 Downvote
That's awesome, thank you for understanding.

Regarding concurrency, I will re-enable it for 3.14.2 with the amends I made regarding the session lock. If your hardware is superior (compared to the DO $5 VPS) then is likely that the process could be faster for your need.

The revision should be released on April, second week. Hope it is not too late.
 
Upvote 0 Downvote
Rodolfo,

Thank you for the reply and continued support.

That will be excellent! In the mean time I will finish prepping my imports so they will be ready.

As for my hardware, I am operating on a SSD SAN and IO is not really an issue as long as it's less than 10 Gbps.

I greatly appreciate the assistance.

Blessings,
Steven Dunn
 
Upvote 0 Downvote
Rodolfo,

I am looking at your upcoming change logs and I have a question regarding this post:

Update, I'm working in continuous importing functionality which will allow to automatic import any given content being uploaded to /importing which may be useful in cases where you want to keep the importing running automatically and save all the hassle of a one-time importing process.

Several use cases can be addressed with automatic importing, like those who want to push images from any given device automatically (it just needs to transmit the images via FTP or anything of that kind). It could be also used to dummy activity in your websites, import very huge image sets, etc. It opens so many possibilities.

The functionality is almost ready and I'm doing the final testings now, adding the UI elements needed.

Hope you like the news,
Cheers.
Click to expand...

The new "continuous importing functionality" will it take into account the metadata.json files uploaded with the images? For example, could I just set my 5.5 million image directory to begin uploading in a constant stream and the site would automatically perform the bulk import? If so, it would save me an enormous number of man hours now and in the future instead of uploading in batches. However, the only issue I could see is if the site tried to import an image and the related json file had not finished uploading...it would import the image without a description/category/etc.

Alternatively, if the "continuous importing functionality" can be turned off/on I could simply upload a batch of files and then toggle it on (or activate the cron). I would not have to leave a browser window open on the Bulk Importer page. This would work too.

A few of my questions and thoughts off the cuff. Also, I am looking forward to this new update!

I appreciate your continued assistance.

Blessings,
Steven Dunn
 
Upvote 0 Downvote
Hi Steven,

Continuous importing parse directories and meta exactly as the previous bulk importer, the difference is that the continuous process never ends, stats can be reseted, and it must be issued via the command line (cron).

The idea of making it continuous is to precisely support large imports, so it can work while you are still transferring the images so you don't have to wait until all the set is at the automatic importing path.

Hope it helps.
 
Upvote 0 Downvote
Rodolfo,

Thank you for the detailed reply.

Excellent! This will be a significant time-saver!

I do have another question. How is the following exception handled within the "continuous importing functionality"?

However, the only issue I could see is if the site [cron] tried to import an image and the related json file had not finished uploading ...it would import the image without a description/category/etc.
Click to expand...

I suppose the cron could check for a related json file and if none exists it skips the album directory and/or image.

On that note, that is another good question. If the cron processes an album, deletes the folder and JSON for the album when processed, and then the running FTP process adds more files to the (now processed/deleted) album directory there won't be any JSON related to the album for the next cron operation.

You have probably already taken all of the above into account. Anyways, I supposed I am just running through the possibilities.

Once again, I appreciate all of your assistance and the answering of my questions!

Blessings,
Steven Dunn
 
Upvote 0 Downvote
I think that is feasible what you describe (missing JSON meta) for which I believe that it should be remarked the need to put the JSON files before everything else. It could be also addressed if the system live monitors these json files (that way you can even update user/album details in the continuous process). I will check that.

About existing content, it adds the content to the matched user and/or album. It checks by username and album name, when no match exists it creates what it needs.
 
Upvote 0 Downvote
You must log in or register to reply here.
Back
Top