• Notice: New feature requests and suggestions won't be considered for Chevereto development at this time. Learn more.

Answered Is it possible to migrate images in local storage to external storage?

bangsters

Network license
Joined
Jul 30, 2017
Messages
103
Likes
12
Points
68
#1
Website URL
https://www.pixelsbin.com

Chevereto version
3.10.18

Description of the issue
Hey Rodolfo. Pixelsbin has been growing and we will be making some changes on environment. We have plenty of space on our local server for images, but we will be switching over to cloud for the images. The reason is we will be deploying redundant/failover servers including database. We experienced some downtime a while back due to fibercut hence we are making the change.


We will be switching to cloud (google, amazon, or ftp). Question is how can we migrate the local images to cloud without breaking the links or associations? Also, if down the road, if our cloud subscription needs to be migrated to a different cloud or ftp, is that possible as well without breaking associations to database or links?

Please advise.


And thanks again for this awesome platform.
 

Rodolfo

Chevereto Guru
Staff member
Joined
Oct 7, 2008
Messages
15,766
Likes
3,997
Points
237
Location
Chevereto HQ
Website
rodolfoberrios.com
#2
It is possible but there's no tool that will do that for you automatically, you have to do it all manually. Basically you need to perform 3 actions:

1. Upload local to remote

For FTP/SFTP you can use the rsync command, for S3 is very easy by following this guide: https://chevereto.com/community/threads/migrate-your-existing-images-to-s3.5388/ No idea if Google, OpenStack or others support a similar sync feature. Key here is to sync (local) /images/* to (remote) /images/* using whatever you can for that matter.

2. Rewrite/redirect old URLS

In your local /images/ folder you will need to place a rule (server based) that will rewrite/redirect your old local URLs to the new remote URL. If you run Apache then you will need to write a rewrite rule in an .htaccess file on the root of your /images/ folder, if you run Nginx you will need to write the server rules for that sub-folder. If you run other kind of server then you will have to find how that server handles server side URL rewriting.

3. Update the database

You must run this query once you add the external storage from your dashboard:
SQL:
UPDATE chv_images SET image_storage_id = <external storage id> WHERE image_storage_id = NULL
You also need to update the stats for that external storage, you only need to set the size (bytes) used in total by all the stuff that you uploaded to the external server.

Hope it helps,
Rodolfo.
 

bangsters

Network license
Joined
Jul 30, 2017
Messages
103
Likes
12
Points
68
#5
Hey @Rodolfo. Just want to let you know that all is working great, I updated the table for the images to reflect the chv_storage_id of the external storage. I did not need to make any redirect urls as they did not change. I migrated all images to cloud via gsutil, and all images are in cloud now. Much much faster with multi regional storage.

However, I did notice that in the settings for external storage, it shows that External Storage is 0KB, and Local is the xxxGB. That is incorrect, as all storage is in external now. If I click search link for local storage it opens external storage, and vice versa it I open search link external storage it opens the local storage
 
Last edited:

Rodolfo

Chevereto Guru
Staff member
Joined
Oct 7, 2008
Messages
15,766
Likes
3,997
Points
237
Location
Chevereto HQ
Website
rodolfoberrios.com
#6
Like I said, you need to update the stats. Since you moved local to remote, you need to reflect that change on the stats table.
 

bangsters

Network license
Joined
Jul 30, 2017
Messages
103
Likes
12
Points
68
#7
Thanks I missed that part. How do I update the stats? Particular table?

Also, any new images rhat get uploaded still reflect as local storage eventhough it is saved in external
 
Last edited:

Rodolfo

Chevereto Guru
Staff member
Joined
Oct 7, 2008
Messages
15,766
Likes
3,997
Points
237
Location
Chevereto HQ
Website
rodolfoberrios.com
#8
chv_storages
1533917035976.png
chv_stats
SQL:
SELECT * from chv_stats where stat_type = 'total'
Column "stat_disk_used"
 

bangsters

Network license
Joined
Jul 30, 2017
Messages
103
Likes
12
Points
68
#9

bee

Client
Joined
Dec 2, 2014
Messages
162
Likes
32
Points
78
Location
Thailand
Website
bee.creative.xxx
#10
cname to storage.googleapis.com
You can do that - but if your site uses SSL you will get mixed content. With Google and OVH storages it's not possible to get CNames working with SSL ... when I'm remember right, only with AWS it will work.
 

bangsters

Network license
Joined
Jul 30, 2017
Messages
103
Likes
12
Points
68
#11
Ahh ok thanks for pointing that out. If I do decide to try it, how can I ge the embed codes to reflect my subdomain that is a cname to storage.googleapis.com?
 

Rodolfo

Chevereto Guru
Staff member
Joined
Oct 7, 2008
Messages
15,766
Likes
3,997
Points
237
Location
Chevereto HQ
Website
rodolfoberrios.com
#12
You can map the external storage to any URL, but you must do all the DNS configuration on your own.

Keep in mind that Chevereto doesn't handle your domain DNS settings. You must refer to Google cloud documentation in how use your own cname for your bucket.
 

bangsters

Network license
Joined
Jul 30, 2017
Messages
103
Likes
12
Points
68
#13
Thank you @Rodolfo!! I figured out a way around it. You are 100% correct that Google Cloud Storage does not do HTTPS when it is accessed via CNAME. But I managed to work around it, with redirect and some modification on the route file.

Thanks again.