Strange AutoModerate

szolnoki · Feb 5, 2022

👉 Run our Container based provisioning and document the settings and context that trigger the alleged bug.

*️⃣ Make sure to test the bug in the Docker infra, let us know whatever the bug happens (or not) there.

▶ Reproduction steps

[Example: Update from X to Y
[Example: Open the website]
[Example: Go to /upload and upload a X image]

😢 Unexpected result

[Example: The website responds with a 500 error, in some routes it is just a white screen.]

📃 Error log message

[Example: Attached is the server error log relevant to this issue]

szolnoki · Feb 5, 2022

Relevant config:
ModearatContent: Enabled
ModearatContent API Key is set
Automatic approve: enabled
Block content: Disabled
Flag NSFW: Teen and adult

Steps:
1. Upload a porn picture as guest

Expected result:
ModerateContent detect as Adult (OK)
Upload accepted, and "We must approve the uploaded content before being able to share." message (OK)
Get a link to image (OK)
Not show as published imege until admin approval (OK)
Image link not working until admin approval
Show for admin Approval

Unexpected result:
Image link working without admin approval
Not show for admin Approval

szolnoki · Feb 5, 2022

Maybe I found the problem root in class.image.php:

Original ccode:

       if (Settings::get('moderatecontent_auto_approve') && isset($image_upload['moderation'])) {
            $values['is_approved'] = 1;^M
        }

As you see, code check only $image_upload['moderation'] varieble exists or no.

But this variable is an structure. Real value example:
stdClass Object ( [url_classified] => multipart/form-data [rating_index] => 3 [rating_letter] => a [predictions] => stdClass Object ( [teen] => 1.0858196765184 [everyone] => 0.18649923149496 [adult] => 98.727679252625 ) [rating_label] => adult [error_code] => 0 )

Need much more complex statement, to check against Settings::get('moderatecontent_block_rating').

Currently, if ModerateContent auto approve is enabled, this statement is absolutely wrong!!!

szolnoki · Feb 5, 2022

Patch is ready, working as expected

--- app/lib/classes/class.image.php.old 2021-12-13 16:22:53.000000000 -0500
+++ app/lib/classes/class.image.php     2022-02-05 10:44:46.934557203 -0500
@@ -1087,7 +1087,8 @@
             break;
         }

-        if (Settings::get('moderatecontent_auto_approve') && isset($image_upload['moderation'])) {
+        if (Settings::get('moderatecontent_auto_approve') && isset($image_upload['moderation']) && isset($image_upload['moderation']->rating_letter)
+                       && $image_upload['moderation']->rating_letter!='a' && $image_upload['moderation']->rating_letter!='e') {
             $values['is_approved'] = 1;
         }

szolnoki · Feb 5, 2022

Ok...
But I have already a problem with police, because anybody uploaded dozens child porn to my Chevereto site
If no automatic moderate, just only one was remain: enable moderation on ALL pictures (private/public)

How can I exclude pron/childporn picture from my site? I don't want asks form police again. 😀
How can I protect my site?

Rodolfo · Feb 5, 2022

szolnoki said:
Patch is ready, working as expected

--- app/lib/classes/class.image.php.old 2021-12-13 16:22:53.000000000 -0500 +++ app/lib/classes/class.image.php 2022-02-05 10:44:46.934557203 -0500 @@ -1087,7 +1087,8 @@ break; } - if (Settings::get('moderatecontent_auto_approve') && isset($image_upload['moderation'])) { + if (Settings::get('moderatecontent_auto_approve') && isset($image_upload['moderation']) && isset($image_upload['moderation']->rating_letter) + && $image_upload['moderation']->rating_letter!='a' && $image_upload['moderation']->rating_letter!='e') { $values['is_approved'] = 1; }

^ It is hard to me to understand what you are trying to tell me here. What is the problem and the alleged solution here?

The moderation check is carried here (ModerateContent class):

PHP:

    private function assertIsAllowed(): void
    {
        $block = [];
        $blockRating = Settings::get('moderatecontent_block_rating');
        switch ($blockRating) {
            case 'a':
                $block[] = 'a';
            break;
            case 't':
                $block[] = 'a';
                $block[] = 't';
            break;
        }
        $ratings = [
            'a' => _s('adult'),
            't' => _s('teen'),
        ];
        foreach ($block as $rating) {
            if ($this->moderation->rating_letter == $rating) {
                throw new Exception(_s('Content of type %s is forbidden', $ratings[$rating]), 1404);
            }
        }
    }

As you may see, it blocks based on the rating previously defined.

szolnoki · Feb 5, 2022

The original problem was:
Bug in class.image.php file, when ModeratorContent is enabled.
The if...then case was buggy. See my message #3 for details.
The patch in my #4 message is the bugfix.

The last problem:
If you will finish the ModeratorContent, how can I protect my Chevereto gallery?

Sorry, my English not a very well.

Rodolfo · Feb 5, 2022

I believe that you are guessing that the problem with moderate content is that my code fails to moderate some images because the IF only checks for an isset, and you assume that the issue is that I don't check letter rating. I do, check the code above (assert function).

Let me emphasize that ModerateContent offers unreliable results, this was already reported by other users. I believe that the problem with moderate content is the service itself, try it out injecting dummy results and you will see that it doesn't allow upload:

app/lib/classes/class.moderatecontent.php

PHP:

public function __construct($imageFilename, $info=[])
    {
        $this->imageFilename = $imageFilename;
        $this->imageInfo = $info;
        if ($this->imageInfo === []) {
            $this->imageInfo = G\get_image_fileinfo($this->imageFilename);
        }
        $this->optimizeImage();
        $url = "http://api.moderatecontent.com/moderate/?key=".Settings::get('moderatecontent_key');
        $this->error_message = '';
        $this->error_code = 0;
        // Dummy response data:
        $curl_response = [
            'url_classified' => 'multipart/form-data',
            'rating_index' => '3',
            'rating_letter' => 'a',
            'predictions' => [
                'teen' => 1.0858196765184,
                'everyone' => 0.18649923149496,
                'adult' => 98.727679252625,
            ],
            'rating_label' => 'adult',
            'error_code' => '0'
            ];
        $curl_response = json_encode($curl_response);
        $json = json_decode($curl_response);
        if ($json === null && json_last_error() !== JSON_ERROR_NONE) {
            $this->error_message = 'Malformed content moderation response';
            $this->error_code = 1403;
        } else {
            $this->moderation = $json;
            $this->assertIsAllowed();
        }
    }

You should get this in the log:

Code:

>> CHV\UploadException [1404]: Content of type adult is forbidden
At /app/lib/classes/class.moderatecontent.php:135

szolnoki · Feb 5, 2022

I know the result 🙂
This exception generated by "$this->assertIsAllowed();".

But I ask again...
Try with BlockContent=Disabled 🙂 Please... 🙂

In this case, you will not get "Content of type adult is forbidden", and upload will be success!!!

But the bug in this case:
You will not find the uploaded image under admin approval, and image share link will be working instantly without approval...

My problem not concern about "blocking"... blocking is working correctly.

Rodolfo · Feb 5, 2022

Why it should block content if the option is disabled?

Block content (disabled) -> Doesn't do anything with the content
Block content (adult) -> Blocks adult
Block content (teen) -> Blocks teen + adult

👉 https://v3-docs.chevereto.com/settings/external-services.html#block-content

szolnoki · Feb 5, 2022

Block content (disabled) -> Doesn't do anything with the content

Yes not blocking... but auto approval need to work.

Ok try explain other ways.
You are always saying "blocking". Ok. This is working. (BlockContent=Adult/Teen)

But I'm saying "auto approval" (BlockContent=Disabled), and this is the original code for auto approval:

        if (Settings::get('moderatecontent_auto_approve') && isset($image_upload['moderation'])) {
            $values['is_approved'] = 1;
        }

This is a statement for "Auto Approval". But this statement wrong, because "isset" will be always true, when AutoApprove=Enabled. This statement not depends from image classification (teen,adult,everyone).

And maybe this is reason, why ModeContent working bad in Chevereto.
I think, Auto approval and blocking is a differenet behavior and different settings.

Another explain:
If the blocking is the only action for "suspicious" images... What is auto Approval? 😀 😀 😀

Currently, I setup two Chevereto V3 site. I will patch only one. I will give you admin passwords for both, and a porn picture.
You will be able test the behaviors.

Rodolfo · Feb 5, 2022

If you have suggestions in how to improve how ModerateContent works in Chevereto there's the RFC section. Please don't use bug reporting for discussion in how something should or shouldn't work.

Strange AutoModerate

szolnoki

Chevereto Member

szolnoki

Chevereto Member

szolnoki

Chevereto Member

szolnoki

Chevereto Member

szolnoki

Chevereto Member

Rodolfo

👑 Chevereto Godlike

szolnoki

Chevereto Member

Rodolfo

👑 Chevereto Godlike

szolnoki

Chevereto Member

Rodolfo

👑 Chevereto Godlike

szolnoki

Chevereto Member

Rodolfo

👑 Chevereto Godlike