• Welcome to the Chevereto user community!

    Here users from all over the world gather around to learn the latest about Chevereto and contribute with ideas to improve the software.

    Please keep in mind:

    • 😌 This community is user driven. Be polite with other users.
    • 👉 Is required to purchase a Chevereto license to participate in this community (doesn't apply to Pre-sales).
    • 💸 Purchase a Pro Subscription to get access to active software support and faster ticket response times.

Physical Memory Usage always increasing.

Status
Not open for further replies.
A

Ankur agarwal

Chevereto Member
Website URL
http://www.gifforall.com

Chevereto version
3.9.5

Description of the issue
Hi,

I purchased network bundle in which I got Network licence and hosting for CHEVERETO,
Script is running fine, Thanks for the chevereto for this.
But problem is with my server. When i upload file via API ( By providing URL in source ), Then its increasing our Physical Memory Usage. I have 1.5 GB allocated Physical Memory but when i upload only 30-40 images then usages goes to 1.2 GB, And it will show RED sign in cpanel.
I think this process is using too much RAM.
We have millions of images, If we go like this then it will take more then 10 years to upload all files.
Please check this issue ASAP.
 
Usually it all depends on your settings. For instance, is not the same job to upload a 10mb vs a 1mb image, apply watermark or not, remove exif tags, etc.

The resizing function is very heavy intensive and it is already optimized. But that can easily fail if your server settings are are wrong or if you are allowing large images to be uploaded, it also depends on how many simultaneous uploads you have, etc.

Bottom line is that you need too have a proper server setup and have a machine for the load you want to throw.
 
Thanks for the update.
Actually image links collected from the internet so dont their size.
No, we are not adding any watermark.
And i am running API upload script on my local server, So i think there is no issue with exit tag.

Server is setup by server guys, Can you please suggest some good server settings??

I am running script in for loop.
For example suppose i have 1000 images in image link database, then i am using this method.
1.) Fetch 30 records from database
2.) API curl in while loop of database row.
 
c_Panel_Main.png

Here is my current server status. While I dont have any traffic on my website.
 
Then do it slower because you are forcing the machine.
 
Depends on you machine, try with 1 second and keep trying.

It is very normal to have issues when in a php while loop if you process too much stuff, be very careful with that thing.
 
OK,
I added 15 second delay in uploading via sleep method, It means now I am uploading only 4 images per minute.
But now just for 4 images physical memory going from 729 MB to 829 MB.

Just for 4 images, I dont think this is good memory consuption.
 
The system doesn't include a loop that process uploads like you are doing. I can't answer you from code that I don't even know. I don't know how optimized your loop is, I don't know how good or bad your server is, I don't know what other process are running.

If you really want help then start by sharing the procedure that you are doing. I need the real code you are using. I can't help you any further if you don't provide more details.

By the way, support is for script issues not custom implementations like this one so please understand that this is outside the scope of support.
 
Hi,
Thanks for your quick reply.

The system doesn't include a loop that process uploads like you are doing.
Yes, I can understand. But I am running loop script( allready attached in this reply ) on my local machine,Not on server. I am just calling api via my localhost php script.
So i dont think this loop script impact on server.

I don't know how good or bad your server is.
Purcahsed TMD network bundle with your hosting package. I think that is good server just for startup.

If you really want help then start by sharing the procedure that you are doing. I need the real code you are using.
Yes, I can understand. Allready shared code file as an attachment.

Please tell how i can send private message here, failed to figure out? or If you want i can show you on teamviewer and talk on skype if you want.
 

Attachments

  • insert_into_gifforall.txt
    2.7 KB · Views: 3
Ankur agarwal said:
Just for 4 images, I dont think this is good memory consuption.
Click to expand...

Image resizing requires lots of memory and that memory consumption depends on how large the image files are. Larger the image gets, larger RAM it eats. Memory consumed by each image resizing is about 10X the size of the file and Chevereto uses native php gd library with already optimized functions. Good as it gets, and Chevereto does way more stuff but the footprint is extremely low.

Based on your feedback, your memory increased 100MB, that could be some of the following:
  • One single 10MB image
  • 10 1MB images
  • 5 2MB images
  • Etc
Bottom line is that every operation has its machine footprint, your while loop exhausted the machine and you should be aware of the fact that in programing there are always limits and you must know before sending instructions because things like this could happen.
 
OK, Understand.

I understand what you are trying to say.
This is happeninig because of large image size.
Yes i am agree with that. May be images are very big in size, Thats why its consuming lots of memory for resizing.
Is this possible we can disable image resizing, so it will not consume lots of RAM. Is this possible?
 
By default the system will always do at least 2 resizing operations. It can't be turned off.

If you aren't getting critical system alerts then don't worry, your importer won't be running indefinitely isn't? Just import what you have left and just try to don't hit a criticality warning.

Since you mentioned it, let me address that our bundles could include shared, private and vps servers. Those are machines totally different, with totally different purposes.
 
Rodolfo said:
By default the system will always do at least 2 resizing operations. It can't be turned off.
Click to expand...
OK.

Rodolfo said:
By the way, why you worry that much about memory consumption? If you aren't getting critical system alerts then don't worry, your importer won't be running indefinitely isn't?
Click to expand...
Showing critical error symbol when Physical Memory Usage goes above 800MB. I am monitring this because i think this is shared hosting.I faced this type of issue 2 years ago.
If my Physical memory Usage go again and again in critical state then they will suspend my hosting plan (because of shared plan in network bundle). This is the problem.
 
Ankur agarwal said:
If my Physical memory Usage go again and again in critical state then they will suspend my hosting plan (because of shared plan in network bundle).
Click to expand...

Our bundles apply for any TMDHosting product, including VPS. Yes, I agree on that most likely shared server is the problem here.
 
The link goes to their default shared hosting plans but any purchase on TMDHosting (after clicking our referral) applies for our Bundles.

I've checked and you purchased a shared server.
 
Unfortunately, I can't help you any further since you have concerns with memory consumption in a shared server. If you want your own dedicated memory then you should get at least a VPS.

My advice is to don't stress your server and complete the import in a way that it doesn't stress your remote machine. That could be achieved with little delays between calls, maybe do the import job in local (where you have plenty RAM) then just upload your local Chevereto to remote, etc.

There are countless workarounds for what are you trying here but this is not the opportunity to talk about it since it doesn't meet the scope of Tech support.

Cheers,
Rodolfo.
 
Status
Not open for further replies.
Back
Top