Problem of adding S3 compatible storage

koxle · Sep 26, 2018

Hi,

I face on a little bit problem on adding Add S3 compatible storage.
There is my step for adding S3 compatible storage(Example: WASABI CLOUD STORAGE).

Step1.

Copy amazon whole folder in app/vendor and rename as wasabi in app/vendor

Step2.

Edit endpoint.json.php in app/vendor/wasabi/Aws/data

Code:

<?php
// This file was auto-generated from sdk-root/src/data/endpoints.json
return [ 'version' => 2, 'endpoints' => [ 'us-west-1/s3' => [ 'endpoint' => 's3.us-west-1.wasabisys.com', ], 'us-east-1/s3' => [ 'endpoint' => 's3.wasabisys.com', ] ],];

Step3.

Edit class.storage.php in app/lib/classes

Code:

protected static $apis = [
        1 => [
            'name'    => 'Amazon S3',
            'type'    => 's3',
            'url'    => 'https://s3.amazonaws.com/',
        ],
        2 => [
            'name'    => 'Google Cloud',
            'type'    => 'gcloud',
            'url'    => 'https://storage.googleapis.com/',
        ],
        5 => [
            'name'    => 'FTP',
            'type'    => 'ftp',
            'url'    => null,
        ],
        6 => [
            'name'     => 'SFTP',
            'type'    => 'sftp',
            'url'    => null,
        ],
        7 => [
            'name'     => 'OpenStack',
            'type'    => 'openstack',
            'url'    => null,
        ],
        8 => [
            'name'     => 'Local',
            'type'    => 'local',
            'url'    => null,
        ],
        9 => [
            'name'  => 'Wasabi S3',
            'type'  => 'wasabi',
            'url'   => 'https://s3.wasabisys.com/',
        ],
    ];

Code:

                    case 's3':
                        $source_file = @fopen($v['file'], 'r');
                        if (!$source_file) {
                            throw new Exception('Failed to open file stream', 100);
                        }
                        $API->putObject([
                            'Bucket'=> $storage['bucket'],
                            'Key'    => $keyprefix . $v['filename'],
                            'Body'    => $source_file,
                            'ACL'    => 'public-read',
                            'CacheControl' => $cache_control
                        ]);
                    break;

Code:

                    case 'wasabi':
                        $source_file = @fopen($v['file'], 'r');
                        if (!$source_file) {
                            throw new Exception('Failed to open file stream', 100);
                        }
                        $API->putObject([
                            'Bucket'=> $storage['bucket'],
                            'Key'   => $keyprefix . $v['filename'],
                            'Body'  => $source_file,
                            'ACL'   => 'public-read',
                            'CacheControl' => $cache_control
                        ]);
                    break;

Code:

            case 's3':
                $API->deleteObject([
                    'Bucket'    => $storage['bucket'],
                    'Key'        => $key
                ]);
            break;

Code:

            case 'wasabi':
                $API->deleteObject([
                    'Bucket'    => $storage['bucket'],
                    'Key'       => $key
                ]);
            break;

Code:

            case 's3':
                require_once(CHV_APP_PATH_LIB_VENDOR . 'amazon/aws-autoloader.php');
                return \Aws\S3\S3Client::factory([
                    'version'    => '2006-03-01',
                    'region' => $storage['region'],
                    'command.params' => ['PathStyle' => true],
                    'credentials' => [
                        'key'         => $storage['key'],
                        'secret'     => $storage['secret'],
                    ],
                    'http'    => [
                        'verify' => CHV_APP_PATH_LIB_VENDOR . 'ca-bundle.crt'
                    ]
                ]);
            break;

Code:

            case 'wasabi':
                require_once(CHV_APP_PATH_LIB_VENDOR . 'wasabi/aws-autoloader.php');
                return \Aws\S3\S3Client::factory([
                    'version'   => '2006-03-01',
                    'region' => $storage['region'],
                    'command.params' => ['PathStyle' => true],
                    'credentials' => [
                        'key'       => $storage['key'],
                        'secret'    => $storage['secret'],
                    ],
                    'http'    => [
                        'verify' => CHV_APP_PATH_LIB_VENDOR . 'ca-bundle.crt'
                    ]
                ]);
            break;

Code:

    public static function getAPIRegions($api)
    {
        $regions = [
            's3' => [
                'us-east-1' => 'US East (N. Virginia)',
                'us-east-2' => 'US East (Ohio)',
                'us-west-1' => 'US West (N. California)',
                'us-west-2'    => 'US West (Oregon)',

                'ca-central-1'    => 'Canada (Central)',

                'ap-south-1'      => 'Asia Pacific (Mumbai)',
                'ap-northeast-2' => 'Asia Pacific (Seoul)',
                'ap-southeast-1' => 'Asia Pacific (Singapore)',
                'ap-southeast-2' => 'Asia Pacific (Sydney)',
                'ap-northeast-1' => 'Asia Pacific (Tokyo)',

                'eu-central-1'    => 'EU (Frankfurt)',
                'eu-west-1'        => 'EU (Ireland)',
                'eu-west-2'        => 'EU (London)',
                'eu-west-3'   => 'EU (Paris)',

                'sa-east-1' => 'South America (Sao Paulo)',
            ],
            'wasabi' => [
                'us-west-1' => 'US West - 1 (Wasabi)',
                'us-east-1' => 'US East - 1 (Wasabi)',
            ]
        ];
        foreach ($regions['s3'] as $k => &$v) {
            $s3_subdomain = 's3' . ($k !== 'us-east-1' ? ('-' . $k) : null);
            $v = [
                'name' => $v,
                'url'    => 'https://'.$s3_subdomain.'.amazonaws.com/'
            ];
        }
        foreach ($regions['wasabi'] as $k => &$v) {
            $s3_subdomain = 's3' . ($k !== 'us-east-1' ? ('.' . $k) : null);
            $v = [
                'name' => $v,
                'url'   => 'https://'.$s3_subdomain.'.wasabisys.com/'
            ];
        }
        return $regions[$api];
    }

Step4.

Edit form_storage_edit.php in app/theme/Peafowl/snippets

Code:

<div id="storage-combo">
    <div data-combo-value="1" class="input-label c7 switch-combo">
        <label for="form-storage-region"><?php _se('Region'); ?></label>
        <select name="form-storage-region" id="form-storage-region" class=" text-input">
            <?php foreach (CHV\Storage::getAPIRegions('s3') as $k => $v) {
            ?>
            <option value="<?php echo $k; ?>" data-url="<?php echo $v['url']; ?>"><?php echo $v['name']; ?></option>
            <?php
        } ?>
        </select>
    </div>
    <div data-combo-value="9" class="input-label c7 switch-combo">
        <label for="form-storage-region"><?php _se('Region'); ?></label>
        <select name="form-storage-region" id="form-storage-region" class=" text-input">
            <?php foreach (CHV\Storage::getAPIRegions('wasabi') as $k => $v) {
            ?>
            <option value="<?php echo $k; ?>" data-url="<?php echo $v['url']; ?>"><?php echo $v['name']; ?></option>
            <?php
        } ?>
        </select>
    </div>
    <div data-combo-value="1 2 9" class="switch-combo">
        <div class="input-label c7">
            <label for="form-storage-bucket">Bucket</label>
            <input type="text" id="form-storage-bucket" name="form-storage-bucket" class="text-input" placeholder="<?php _se('Storage bucket') ?>" required>
        </div>
    </div>
    <div data-combo-value="1 9" class="switch-combo">
        <div class="input-label c7">
            <label for="form-storage-key"><?php _se('Key'); ?></label>
            <input type="text" id="form-storage-key" name="form-storage-key" class="text-input" placeholder="<?php _se('Storage key') ?>" required>
        </div>
        <div class="input-label c7">
            <label for="form-storage-secret"><?php _se('Secret'); ?></label>
            <input type="text" id="form-storage-secret" name="form-storage-secret" class="text-input" placeholder="<?php _se('Storage secret') ?>" required>
        </div>
    </div>
    <div data-combo-value="2" class="switch-combo soft-hidden">
        <div class="input-label">
            <div class="c7"><label for="form-storage-key"><?php _se('Client email'); ?></label>
            <input type="text" id="form-storage-key" name="form-storage-key" class="text-input" placeholder="<?php _se('Google Cloud client email') ?>" required></div>
            <div class="input-below"><?php _se('You will need a <a %s>service account</a> for this.', 'href="https://cloud.google.com/storage/docs/authentication#service_accounts" target="_blank"'); ?></div>
        </div>
        <div class="input-label c15">
            <label for="form-storage-secret"><?php _se('Private key'); ?></label>
            <textarea id="form-storage-secret" name="form-storage-secret" class="text-input" placeholder="<?php _se('Google Cloud JSON key') ?>" required></textarea>
        </div>
    </div>
    <div data-combo-value="7" class="switch-combo soft-hidden">
        <div class="input-label c7">
            <label for="form-storage-service"><?php _se('Service name'); ?></label>
            <input type="text" id="form-storage-service" name="form-storage-service" class="text-input" placeholder="swift">
        </div>
        <div class="input-label c7">
            <label for="form-storage-server"><?php _se('Identity URL'); ?></label>
            <input type="text" id="form-storage-server" name="form-storage-server" class="text-input" placeholder="<?php _se('Identity API endpoint') ?>" required rel="template-tooltip" data-tiptip="right" data-title="<?php _se('API endpoint for OpenStack identity'); ?>">
        </div>
        <div class="input-label c7">
            <label for="form-storage-key"><?php _se('Username'); ?></label>
            <input type="text" id="form-storage-key" name="form-storage-key" class="text-input" placeholder="<?php _se('Username') ?>" required>
        </div>
        <div class="input-label c7">
            <label for="form-storage-secret"><?php _se('Password'); ?></label>
            <input type="text" id="form-storage-secret" name="form-storage-secret" class="text-input" placeholder="<?php _se('Password') ?>" required>
        </div>
        <div class="input-label c7">
            <label for="form-storage-region"><?php _se('Region'); ?></label>
            <input type="text" id="form-storage-region" name="form-storage-region" class="text-input" placeholder="<?php _se('Storage region') ?>">
        </div>
        <div class="input-label c7">
            <label for="form-storage-bucket"><?php _se('Container'); ?></label>
            <input type="text" id="form-storage-bucket" name="form-storage-bucket" class="text-input" placeholder="<?php _se('Storage container') ?>" required>
        </div>
        <div class="input-label c7">
            <label for="form-storage-account_id"><?php _se('Tenant id'); ?> <span class="optional"><?php _se('optional'); ?></span></label>
            <input type="text" id="form-storage-account_id" name="form-storage-account_id" class="text-input" placeholder="<?php _se('Tenant id (account id)') ?>">
        </div>
        <div class="input-label c7">
            <label for="form-storage-account_name"><?php _se('Tenant name'); ?> <span class="optional"><?php _se('optional'); ?></span></label>
            <input type="text" id="form-storage-account_name" name="form-storage-account_name" class="text-input" placeholder="<?php _se('Tenant name (account name)') ?>">
        </div>
    </div>
    <div data-combo-value="5 6" class="switch-combo soft-hidden">
        <div class="input-label c7">
            <label for="form-storage-server"><?php _se('Server'); ?></label>
            <input type="text" id="form-storage-server" name="form-storage-server" class="text-input" placeholder="<?php _se('Server') ?>" required rel="template-tooltip" data-tiptip="right" data-title="<?php _se('Hostname or IP of the storage server'); ?>">
        </div>
        <div class="input-label c7">
            <label for="form-storage-bucket"><?php _se('Path'); ?></label>
            <input type="text" id="form-storage-bucket" name="form-storage-bucket" class="text-input" placeholder="<?php _se('Server path') ?>" required rel="template-tooltip" data-tiptip="right" data-title="<?php _se('Server path where the files will be stored'); ?>">
        </div>
        <div class="input-label c7">
            <label for="form-storage-key"><?php _se('Username'); ?></label>
            <input type="text" id="form-storage-key" name="form-storage-key" class="text-input" placeholder="<?php _se('Server username') ?>" required>
        </div>
        <div class="input-label c7">
            <label for="form-storage-secret"><?php _se('Password'); ?></label>
            <input type="text" id="form-storage-secret" name="form-storage-secret" class="text-input" placeholder="<?php _se('Server password') ?>" required>
        </div>
    </div>
    <div data-combo-value="8" class="switch-combo soft-hidden">
        <div class="input-label c7">
            <label for="form-storage-bucket"><?php _se('Path'); ?></label>
            <input type="text" id="form-storage-bucket" name="form-storage-bucket" class="text-input" placeholder="<?php _se('Local path') ?>" required rel="template-tooltip" data-tiptip="right" data-title="<?php _se('Local path where the files will be stored'); ?>">
        </div>
    </div>
    <div class="input-label">
        <div class="c7">
            <label for="form-storage-capacity"><?php _se('Storage capacity'); ?></label>
            <input type="text" id="form-storage-capacity" name="form-storage-capacity" class="text-input" placeholder="<?php _se('Example: 20 GB, 1 TB, etc.') ?>">
        </div>
        <div class="input-below"><?php _se('This storage will be disabled when it reach this capacity. Leave it blank or zero for no limit.'); ?></div>
    </div>
    <div data-combo-value="9" class="switch-combo soft-hidden">
        <div class="input-label">
        <label for="form-storage-url">URL</label>
        <input type="text" id="form-storage-url" name="form-storage-url" class="text-input" placeholder="<?php _se('Storage URL') ?>" value="<?php echo CHV\Storage::getAPIRegions('wasabi')['us-east-1']['url'] ?>" required>
        <div class="input-below"><?php _se('The system will map the images of this storage to this URL.'); ?></div>
    </div>
    </div>
    <div data-combo-value="1 2 3 4 5 6 7 8" class="switch-combo soft-hidden">
    <div class="input-label">
        <label for="form-storage-url">URL</label>
        <input type="text" id="form-storage-url" name="form-storage-url" class="text-input" placeholder="<?php _se('Storage URL') ?>" value="<?php echo CHV\Storage::getAPIRegions('s3')['us-east-1']['url'] ?>" required>
        <div class="input-below"><?php _se('The system will map the images of this storage to this URL.'); ?></div>
    </div>
</div>
</div>

Step5.

Edit your database add a new row to 'chv_storage_apis'
storage_api_id = 9
storage_api_name = Wasabi S3
storage_api_type = wasabi

//-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------//

After this modifying, I can upload images with the new s3 comp. storage,
but I Found that after a click to delete the image from Chevereto script.
It can't be deleted from the bucket. the image is still removed from the database.

I have no idea why it can be upload but can't be deleted from the bucket.
Could anyone tell me why?

Extra: If I just edit the endpoint.json.php in the origin amazon folder and replace the us-east-1 or us-west-1 endpoint link, choose Amazon S3 in the dashboard,
select the region which is modified the endpoint and enter key id and key secret. It can work normally both upload and delete function from the bucket.

Rodolfo · Sep 26, 2018

According to their docs, you only need to touch the endpoint https://wasabi-support.zendesk.com/...How-do-I-use-the-AWS-SDK-for-PHP-with-Wasabi- but it seems to be a complete hassle to achieve that in Chevereto.

I will try to find a way to make Chevereto S3 universal capable.

koxle · Sep 26, 2018

Or try to make endpoint.json.php dynamic change the return value by selecting the profile in Chevereto?
According to the profile which is selected in Chevereto, the form_storage_edit.php will generate the new list of the region.
^^^
It only suitable for activating only one s3 comp. storage. Failed.

Rodolfo · Sep 26, 2018

I'm going to definitely add this for V3.12.0 as it will boost the number of services capable of handling our storage. I will do it in a way it uses the AWS PHP SDK but allows you to customize endpoint+region so you can truly use any S3 compatible provider.

pburel · Jan 29, 2019

I can't enter an S3 compatible endpoint ... bad format of url ... ?

I use a personal https://minio.io/ self hosted storage ... with https://[my IP address]:[port] format ... as endpoint

During the submit process https:[my server]/dashboard/settings/external-storage ... I get an error about the Endpoint field format ...

Any suggestion ?

pburel · Jan 29, 2019

Ok I saw pattern="http://.*" in your form ...my storage is http for now ...

Rodolfo · Jan 29, 2019

You can indicate if the url goes https or not, is a checkbox in the storage listing.

dydyt · Oct 17, 2019

I tried to add Wasabi bucket through "S3 compatible" but it doesn't work. Any tips?

Rodolfo · Oct 17, 2019

This guide is not longer needed because Chevereto v3.12.0 added "S3 compatible" storages.

https://chevereto.com/releases#v3_12_0

@dydyt Please open a ticket indicating the problems you are experiencing with Wasabi.

Problem of adding S3 compatible storage

koxle

Chevereto Member

Rodolfo

⭐ Chevereto Godlike

koxle

Chevereto Member

Rodolfo

⭐ Chevereto Godlike

pburel

Chevereto Member

pburel

Chevereto Member

Rodolfo

⭐ Chevereto Godlike

dydyt

Chevereto Member

Rodolfo

⭐ Chevereto Godlike