S3 storage not saving to folder

[Pro]

Hi,

We originally used local storage which saved all images to /images/. We use a CDN with this so a bunch of URLs exist in the wild like sub.cdn.com/images/date/file.ext

Now we are migrating local to S3 with the same CDN so our images don't break. To do this I had to migrate everything to the S3 directory /images/ in the bucket. These images work, but new images do not upload to the correct location.

This is the way S3 is setup on Chevereto:

Bucket:
sub.cdn.com [.'s are necessary in bucket name due to CDN, anyway this is valid]

URL:
sub.cdn.com/images/ [/images is necessary in this URL otherwise all old images break]

Now the connection works, except even though the URL is mapped to /images/, it uploads in the bucket root (so it is bucket/year instead of bucket/images/year)

I have tried:

- Changing the bucket policy so it only allows access to the images, which gives permission error that says it can't write to bucket root
- Changing the "Bucket" setting above to the same as the "URL" (without trailing slash) which gives error with %2F instead of /

How do I get chevereto to upload to the /images/ directory in my bucket?

Thank you

 

[Rodolfo]

Hi,

- Changing the bucket policy so it only allows access to the images, which gives permission error that says it can't write to bucket root

Chevereto isn't aware of that error, so it won't do anything about it. It will just throw an exception.

- Changing the "Bucket" setting above to the same as the "URL" (without trailing slash) which gives error with %2F instead of /

The bucket needs to be passed to the S3 API, meaning that it literally tries to find the bucket "bucket%2F". A bucket is not the same as a file system path, is just a name.

How do I get chevereto to upload to the /images/ directory in my bucket?

External storage stores files to bucket/$destination where $destination replaces the setting for image path. This means that it won't upload to bucket/images, but directly to bucket/ or to bucket/$datetime.

If for some reason you must use just one bucket and CDN, you should try with a rewrite rule. With CloudFlare you could attach a simple page rule for this purpose.

• Old content: Forward *sub.cdn.com/images/* to https://sub.cdn.com/$2
• New content use sub.cdn.com directly at bucket
• All images are at bucket/date, no images/ needed

Another workaround, have two CDN domains:

• 1 for the old content: sub.cdn.com/images/date/file.ext pointed to the bucket where to locate the old local images under bucket-for-old/images/date.
• 1 for the new S3 content: sub-2.cdn.com/date/file.ext pointed to a bucket for the new uploads, where the images sit at bucket-for-new/date/.

Hope it helps.

 

[Pro]

Thanks, for consistency just wanted to avoid having multiple domains just yet and especially removing the directory. I don't think it's worth redirecting everything though so we'll just use the second subdomain.

I set it up and it's uploading the files, but all of a sudden the watermark is no longer showing. Here is a sample from a new User-level account I made:

car 2 1

Image car 2 1 in Test's images album

www.hostmyjunk.com

You can see basically all the previous images for comparison.

Here are the watermark settings:

2020-10-28_09-55-29.png

Shared with Dropbox

www.dropbox.com

Edit: I tried enabling and disabling watermarks, no change.

 

[Pro]

Also I don't know if related, but I am getting this error when trying to do uploads with duplicate filenames: https://www.dropbox.com/s/es1f4ni6k1yth6x/2020-10-28_10-00-12.png?dl=0

But, notice how that file name is car-2 and the only file I actually uploaded was car-2-1 (in the link in my previous post)

Which I thought based on your answer here that it should just automatically rename it, or do I need to select Original + Random in file name settings?

 

[Rodolfo]

Edit: I tried enabling and disabling watermarks, no change.

Watermarks only works for new uploads with the setting enabled. It won't watermark previously uploaded content.

Also I don't know if related, but I am getting this error when trying to do uploads with duplicate filenames: https://www.dropbox.com/s/es1f4ni6k1yth6x/2020-10-28_10-00-12.png?dl=0

Duplicate upload detection works detecting the file hash, not the filename. https://v3-docs.chevereto.com/settings/image-upload.html#enable-duplicate-uploads and is triggered when a non admin account uploads the same image twice.

 

[Pro]

Watermarks only works for new uploads with the setting enabled. It won't watermark previously uploaded content.

Yes this is what I am talking about. Watermarks are not working since switching the storage to S3.

In uploading new images, the watermark is not there.

My comment you quoted was saying "i tried turning on/off watermarks (like rebooting a computer) and it didn't fix it".

 

[Pro]

It works again, strange. Okay, all the issues I've mentioned until now are resolved, thank you.

Although, if I can use this ticket for another thing, we still have that issue from long ago of randomly having a black background on the Watermark. We troubleshooted this some time ago (not sure if you can see old ticket history) but without resolution. I'm wondering if you have any insight into it now that it's still happening? I've spoken to my host (liquidweb) about it at length and they have no idea.

Here are 2 images uploaded in close succession that show the difference, and there are plenty of other examples:

Loading…

j.hmjimg.com

Loading…

j.hmjimg.com

 

[Pro]

I found the old reply from my host about this:

Per our phone conversation your image downloading issue appears resolved. The software was stuck on a particular image and when you marked that one as completed it continued where it left off properly.

Regarding the issue with GD and watermarks this does not appear to be an issue with the server in any way. There are no settings for GD to configure, it is either installed or it is not. GD had been installed since 2016 without issue however as a troubleshooting step we did recompile PHP which reinstalled GD and it did not help. From the forum post you provided it looks like this is unrelated to any server settings on our end as a third party stated the following:

Although I don't use a watermark I was curious so I thought I'd give this a try on my test site. It runs on cPanel with php 7.2.7 and GD bundled 2.1.0 compatible.

The image attached above in this thread gave a black blackground. I then created my own watermark using Photoshop, that did not produce a black background and worked ok.

The fact that he got a black background under some circumstances shows that it is happening outside of your server as well. The developer then stated the following:

Could be that the watermark fails when the image is too complex. Maybe is just a limitation of the implementation.

This is them saying it could be a limitation of the implementation of the watermark, which we could not assist with as that is an issue with the software involved whether it be GD or the watermark software itself.

We could try upgrading you to EA4 which would come with a higher version of GD but we cannot guarantee it would help. If the developer is unable to provide a solution I would recommend trying a different watermark solution.

If the developer recommends any server changes be made to troubleshoot this however we will be happy to help, just let us know what settings you would like changed.

We are now on the latest version of PHP available via cPanel (7.4), GD continues to work fine on other sites on the server.

 

[Rodolfo]

Here are 2 images uploaded in close succession that show the difference, and there are plenty of other examples:

https://j.hmjimg.com/2020/10/28/post-222-1598750690-002.jpg https://j.hmjimg.com/2020/10/28/post-222-1598492604_thumb-002.jpg

That kind of behavior will be always because GD library, which varies a lot and that's one of the reasons why I want to drop it in favor or imagemagick.

For now you will have to tolerate the GD library annoyances.

 

[Pro]

That kind of behavior will be always because GD library, which varies a lot and that's one of the reasons why I want to drop it in favor or imagemagick.

For now you will have to tolerate the GD library annoyances.

Ok, at least it's a known issue.

Is i(mage)magick on the roadmap?

 

[Rodolfo]

Ok, at least it's a known issue.

Is i(mage)magick on the roadmap?

Work in progress - Chevereto v3.18 announcement

Hey there, I'm trying something new here so I will start announcing multiple upcoming releases. That way I think that we can have more time to discuss the changes, allowing me to build better releases. I'm announcing Chevereto v3.18 which will focus in improve the image processing system...

chevereto.com

Yes, just now.

 

[Pro]

Good news, thank you!