Social+ Docs
About UsDeveloper PortalArchives
  • Social+
  • Announcements
  • Technical FAQ
  • Getting Started
    • Installation and Authentication
      • Install iOS SDK
      • Install Android SDK
      • Install JavaScript SDK (Deprecated)
      • Install TypeScript SDK
      • Install Flutter SDK
      • Install SDK for Ionic
    • Tutorials
  • Social+ SDK
    • Core Concepts
      • Session State
      • User
        • Create User
        • Get User Information
        • Search and Query Users
        • Update User Information
        • Delete User
        • Roles & Permissions
        • User Token Management
        • Flag / Unflag User
      • Files, Images, and Videos
        • File Handling
        • Image Handling
        • Video Handling
      • Push Notifications
        • iOS Push Notification Certificate Setup
        • Android Push Notifications Initialization
        • React Native Push Notifications Initialization
        • Register and Unregister Push Notifications on a Device
        • Push Notification Toggles
          • User Push Notification Settings
          • Channel Push Notification Settings
          • Community Push Notification Settings
      • Mentions
      • Poll
      • Ads
        • Get Ads and Settings
        • Ad Impressions
      • Presence State
        • Heartbeat Sync
        • User Presence
        • Channel Presence
      • Live Objects/Collections
        • iOS Live Objects/Collections
        • Android Live Objects/Collections
        • Javascript Live Objects/Collections
        • TypeScript Live Objects/Collections
        • Flutter Live Objects/Collections
      • Realtime Events
        • Social Realtime Events
        • Chat Realtime Events
      • Error Handling
      • Content Moderation
      • Logging
    • Chat
      • Channels
        • Create Channel
        • Query Channels
        • Get Channel
        • Get Channels
        • Update Channel
        • Join/Leave Channel
        • Subchannel
        • Message Preview
        • Unread Count
          • Channel Unread Count
          • Start / stop message read receipt sync
          • Mark Message as Read
          • Message Delivery Status
          • Observe reading count
        • Read Status & Unread Count (Legacy)
        • Channel Moderation
        • Query Members
        • Search Members
        • Archive Channels
      • Messaging
        • Send a Message
          • Text Message
          • Image Message
          • Audio Message
          • Video Message
          • File Message
          • Custom Message
          • Reply to a Message
        • Query and Filter Messages
        • Get and View a Message
        • Edit and Delete Messages
        • Message Reaction
        • Mention in Messages
        • Flag/Unflag a Message
      • Moderation
        • Roles and Permission
        • Mute/Unmute a List of Channel Members
        • Ban/Unban a List of Channel Members
        • Channel Rate-Limiting
    • Social
      • Posts
        • Create Post
          • Text Post
          • Image Post
          • File Post
          • Video Post
          • Live stream Post
          • Poll Post
          • Custom Post
        • Query Post
        • Get Post
        • Viewing Post Content
        • Edit Post
        • Delete Post
        • Mention in Post
        • Flag / Unflag Post
        • Post Review
        • Post Impression
        • Pinned Post
      • Stories
        • Get Global Story Targets
        • Create Story
        • Get Stories
        • Delete Story
        • Get Story Targets
        • Story Impressions
      • Comments
        • Create Comment
        • Query Comment
        • View Comment
        • Get Comment
        • Get Latest Comment
        • Edit Comment
        • Delete Comment
        • Flag/Unflag
        • Get Comment Reaction Data
        • Mention in Comment
      • Reactions
        • Query Reactions
        • Add / Remove Reaction
      • Feed
        • Custom Post Ranking
        • Query Global Feed
      • Communities
        • Create Community
        • Get Community
        • Query Communities
        • Update Community
        • Delete Community
        • Join/Leave Community
        • Trending and Recommended Communities
        • Query Community Members
        • Community Moderation
        • Community Categories
      • Follow/Unfollow
        • Get Connection Status and Connection Counter
        • Follow/Unfollow User
        • Get Follower/Following List
        • Accept/Decline Follow Request
      • Block and Unblock User
      • Intelligent Search
        • Intelligent Search - Post
        • Intelligent Search - Community
      • Notification tray
        • Get Notification Tray Seen
        • Mark Notification Tray Seen
        • Query Notification Tray Item
        • Mark Notification Tray Item Seen
    • Video
      • iOS
        • View & Play Live Stream
        • Broadcast Live Stream
        • Push Notifications
      • Android
        • View & Play Live Stream
        • Broadcast Live Stream
        • Push Notifications
      • Flutter
        • View & Play Live Stream
        • Push Notifications
      • Web
        • View & Play Live Stream
        • Create Live Stream
        • Push Notifications
      • TypeScript (Beta)
        • RunQuery Pattern
        • Live Stream
      • React Native
        • View & Play Live Stream
        • Broadcast Live Stream
    • Changelogs & Versioning
      • iOS
      • Android
      • TypeScript
      • React Native
      • Flutter (Beta)
      • React Native (Video)
      • JavaScript (Deprecated)
  • Social+ UIKit
    • UIKit 4
      • Installation Guide
        • iOS
        • Android
        • Web React
        • Flutter (Beta)
        • React Native (Beta)
      • Setup and Authentication
        • iOS
        • Android
        • Web React
        • Flutter (Beta)
        • React Native (Beta)
      • Customization
        • Remote Config
        • Customization Basics
        • Overriding Navigation Behaviour
      • Social
        • Discover & Search
          • Social Home Page
            • Top Navigation Component
            • Newsfeed Component
            • Empty Newsfeed Component
            • Global Feed Component
            • Create Post Menu Component
            • Community Categories Component
            • Recommended Communities Component
            • Trending Communities Component
          • Global Search Page
            • Top Search Bar Component
            • Community Search Result Component
            • User Search Result Component
          • My Communities Search Page
          • All Categories Page
          • Communities By Category Page
          • Notification Tray
        • Community
          • Community Setup Page
            • Add Category Page
            • Add Member Page
          • Community Profile Page
            • Community Header Component
            • Community Feed Component
            • Community Pin Feed Component
            • Community Image Feed Component
            • Community Video Feed Component
          • Pending Post Page
            • Pending Post Content Component
          • Community Setting Page
            • Community Notification Setting Page
              • Community Posts Notification Page
              • Community Comments Notification Page
              • Community Stories Notification Page
            • Community Post Permissions Page
            • Community Story Setting Page
          • Community Membership Page
        • User
          • User Profile Page
            • User Profile Header Component
            • User Feed Component
            • User Image Feed Component
            • User Video Feed Component
          • Edit User Profile Page
          • Blocked User Page
          • User Relationship Page
          • User Pending Follow Request Page
        • Post
          • Post Details Page
            • Post Content Component
          • Post Composer Page
            • Media Attachment Component
            • Detailed Media Attachment Component
          • Post Target Selection Page
          • Poll Post Composer Page
          • Poll Target Selection Page
        • Livestream
          • Livestream Target Selection Page
          • Livestream Creation Page
          • Livestream Player Page
          • Livestream Terminated Page
        • Story
          • Story Target Tab Component
          • Story Creation Page
            • Story Drafting Page
          • Story Viewing Page
          • Story Target Selection Page
        • Comment & Reaction
          • Comment Tray Component
          • Reaction List Component
        • Content moderation
      • Chat
        • Recent chats (channel list)
          • Chat Home Page
          • Chat List Component
          • Archived Chat List Component
          • Amity Channel Create Conversation Page
        • Conversation Chat
          • AmityChatPage
          • AmityMessageComposer
        • Live Chat
          • Live Chat Page
            • Live Chat Header Component
            • Live Chat Message List Component
              • Reaction List Component
            • Live Chat Compose Bar Component
    • UIKit 3
      • iOS (Deprecated)
        • Overview
        • iOS UIKit Installation Guide
        • Installing UIKit (deprecated)
        • Setup & Authentication
        • Using Themes
        • Social UIKit
          • Using as a whole feature with the default settings
          • Our Components
            • Community Home Page
              • Newsfeed
                • Global Feed
                • My Community Preview
              • Explore Tab
                • Recommended Community
                • Top Trending
                • Categories
            • My Community Page
            • Category List Page
            • Community List by Category Page
            • Community Creation Page
            • Community Profile Edit Page
            • Community Setting Page
              • Community Push Notification Settings Page
                • Community Push Notification Settings - Comment and Reply Relate Event Page
                • Community Push Notification Settings - Post Relate Event Page
            • Post Creation Page
              • Livestream Post
            • Post Detail Page
            • Post Edit Page
            • Comment Creation
            • Comment Edit Page
            • Community Profile Page
              • Community Feed
              • Media Gallery
            • Community Member Page
            • User Profile Page
              • User Feed
            • User Profile Page (2.2)
              • User Feed (2.2)
            • User Setting Page(2.2)
            • User Profile Edit Page
            • Following / Follower User List Page (2.2)
            • Follow Request Page (2.2)
            • Member Selection Page
            • Post Target Selection Page
          • Using Your Own Component
            • Using Your Own Navigation Bar
          • Overriding UIKit Behaviour
            • Feed UI Settings
              • Post Sharing
              • Post Rendering
            • Event Handling
              • Example
          • Roles & Permissions
        • Chat UIKit
          • Using as a Whole Feature with the Default Settings
          • Using Only Some Components
            • Chat Home Page
              • Recent chat
            • Chat Room Page
          • Using your own component
            • Message Bubble
        • Video UIKit
          • Livestream
      • Android (Deprecated)
        • Overview
        • Android UIKit Installation Guide
        • Installing UIKit (deprecated)
        • Setup & Authentication
        • Using Themes
        • Social UIKit
          • Using as a whole feature with the default settings
          • Our Components
            • Community Home Page
              • Newsfeed
                • Global Feed
                • My Community Preview
              • Explore Tab
                • Recommended Community
                • Top Trending Community
                • Categories
            • My Community Page
            • Category List Page
            • Community List By Category Page
            • Community Creation Page
            • Community Profile Edit Page
            • Post Creation Page
              • Livestream Post
            • Post Details Page
            • Post Edit Page
            • Comment Creation
            • Comment Edit Page
            • Community Profile Page (2.5)
              • Community Feed (2.5)
              • Media Gallery
            • Community Member Page
            • User Profile Page
              • User Feed
            • User Profile Page (2.3)
              • User Feed (2.3)
            • User Setting Page (2.3)
            • Following and Follower User List Page(2.3)
            • Follow Request Page (2.3)
            • User Profile Edit Page
            • Member Selection Page
            • Post Target Selection Page
          • Using Your Own Components
            • Using Your Own Navigation Bar
          • Overriding UIKit Behaviour
            • Overriding Global Behaviour
              • Feed UI Settings
                • Post Sharing
                • Post Rendering
            • Event Handling
              • Example
        • Chat UIKit
          • Using as a whole feature with the default settings
          • Using Only Some Components
            • Chat Home Page
              • Recent Chat
            • Chatroom Page
          • Using Your Own Component
            • Message Bubble
      • React (Deprecated)
        • Overview
        • Web UIKit Installation Guide
        • Installing UI Kit (deprecated)
        • Setup & Authentication
        • Using Themes
        • Social UIKit
          • Using as a whole feature with the default settings
          • Our Components
            • Community Home Page
              • Search Communities
              • Newsfeed
                • Global Feed
              • Explore
                • Recommended Communities
                • Top Trending Communities
                • Categories
            • My Community
            • Community List By Category Page
            • Community Creation Page
            • Community Settings
              • Edit Profile Page
              • Members
              • Permissions
            • Post Creation
              • Livestream Post
            • Post Edit
            • Comments
            • Community Profile Page
              • Community Feed
              • Media Gallery
              • Community Members Page
              • Transfer Moderator Role
            • User Profile
              • User Feed
              • Media Gallery
              • Following / Follower User List Page
            • Follow Request Page
            • Profile Settings Page
          • Using Your Own Component
            • Using Your Own Like Button for Posts
          • Overriding UIKit Behaviour
            • Post Rendering
        • Chat UIKit
          • Using as a Whole Feature with the Default Settings
          • Our Components
            • Recent Chat
            • Chat Room
            • Message Bubble
            • Chat Details
        • Event Handling
          • Example
        • Roles & Permissions
      • Flutter (Beta)
        • Overview
        • Flutter UIKit Installation Guide
        • Setup & Authentication
        • Using Themes
        • Social UIKit
          • Using as a whole feature with the default settings
      • React Native (Beta)
        • Setup & Authentication
        • Running the Sample App with Expo
    • Changelogs
      • iOS
      • Android
      • Flutter
      • Web (React)
      • React Native
        • React Native Chat UIKit
        • React Native Social UIKit
  • Analytics & Moderation
    • Social+ Portal
      • Getting Started
      • Application Management
      • Account Management
      • Dashboard
        • Raw Data Export
      • Changelogs
    • Social+ Console
      • Moderation, Roles & Privileges
      • Chat Management
        • Channel Management
        • Message Management
        • Chat Activities (Beta)
      • Social Management
        • Communities
        • Posts
          • Post Viewing and Management
          • Post Creation
          • Post Pinning and Featuring
        • Comments
          • Comment Viewing and Management
          • Comment Creation and Reply
        • Stories
      • User & Content Management
        • User Social History
      • Admin Access Control
      • Livestream Moderation
      • Premium Ads
        • Setting Up Premium Ads
        • Setting Up Advertiser Profile
      • Settings
        • Image Moderation
        • Push Notifications
        • Security
          • Admin Token Management
        • Brand Settings
      • AI Content Moderation
      • Changelogs
    • Social+ APIs and Services
      • Network Settings
      • Generate User Last Activity Report
      • Pre-Hook Event
  • Developers
    • Developer Kits
    • Beta Features
      • Enhance Moderation
      • Search Posts API
      • Content Search V2 (Deprecated)
      • Webhook Events
      • Notification Tray
      • Block User
      • User Activity
    • Migration Guides
      • FCM legacy API Migration Guide
      • SDK v6.26.0 Unread Count Migration Guide
      • JS SDK to TS SDK Migration Guide
      • Web UIKit v3.0 Migration Guide
      • iOS UIKit V3.0 Migration Guide
      • iOS SDK v7.0.0 Migration Guide
      • iOS SDK v6.0.0 Migration Guide
        • iOS SDK v6.9.0 Deprecation Note
        • iOS SDK v6.8.0 Deprecation Note
        • iOS SDK v6.7.0 Deprecation Note
        • iOS SDK v6.5.0 Deprecation Note
      • Android SDK v7.0.0 Migration Guide
      • Android SDK v6.0.0 Migration Guide
        • Android SDK v.6.20.0 Deprecation Note
        • Android SDK v.6.19.0 Deprecation Note
        • Android SDK v6.10.0 Deprecation Note
        • Android SDK v6.9.0 Deprecation Note
        • Android SDK v6.5.0 Deprecation Note
      • TS SDK v7.0.0 Migration Guide
      • TS SDK v6.0.0 Migration Guide
        • TS SDK v6.4.6 Deprecation Note
        • TS SDK v6.8.0 Deprecation Note
      • Web SDK v5.0.0 Migration Guide
      • Flutter SDK v7.0.0 Migration Guide
      • Flutter SDK beta Migration Guide
        • Flutter SDK v0.21.0 Deprecation Note
    • SDK API Reference
      • Server API Reference
      • iOS SDK API Reference (beta)
      • Android SDK API Reference (beta)
      • Flutter SDK API Reference
  • Miscellaneous
    • Support
      • Code of conduct
      • Managed UI Kit - Deprecation
      • Contribute
      • Security
    • Announcements Archive
Powered by GitBook
On this page

Was this helpful?

Export as PDF
  1. Analytics & Moderation
  2. Social+ Console

AI Content Moderation

Creating safer communities for all

Last updated 12 days ago

Was this helpful?

At Social Plus, we leverage AI for automatic content moderation to ensure that you have a safe online environment. We offer two types of AI moderation:

  • Pre-Moderation: Content is automatically reviewed before it is posted. Our AI system scans the content upon upload and generates a confidence value. If this confidence value is equal to, or higher than the configured threshold, the content will be blocked from being posted.

  • Post-Moderation: Post-moderation occurs after content has been posted. Our AI system considers two main factors, flagConfidence and blockConfidence. It scans the content posted and generates a confidence value. Based on this value, different actions are automatically taken - depending on the configured threshold.

Enabling AI Content Moderation

To enable the AI Content Moderation feature, please contact our . Once the support team raises a task ticket, our development team will enable the feature for you.

AI Pre-Moderation

Our AI pre-moderation feature is currently only available for image moderation. It ensures that all uploaded images are scanned for inappropriate, offensive, and undesirable content before it is published. Our AI system scans and detects undesirable content in images across 4 categories:

  • Nudity

  • Suggestive content

  • Violence

  • Disturbing

To enable image moderation, log in to Social Plus Console. Under Moderation > Image Moderation, toggle "Enable image moderation" to "ON".

Once you've enabled image moderation, you will need to set the confidence level for each moderation category. In the context of content moderation, confidence levels represent the degree of certainty the AI system has in identifying specific content categories within an uploaded image. It is crucial to set confidence levels for each moderation category to fine-tune the system's sensitivity in a way that meets your moderation needs.

By default, confidence levels are set to "0" for each category. A confidence level of "0" implies a low threshold, making the system more likely to block images, potentially resulting in false positives, even if the content is not inappropriate. For more accurate and reliable results, it is recommended to set confidence levels at a higher threshold. A higher confidence value indicates a stronger certainty in the content classification.

When enabled, our AI system will scan images uploaded in posts, comments, and messages and return a confidence value. If this confidence value is equal to, or higher than the configured threshold, the content will be blocked from being posted. The undesirable image will need to be removed by the user for their post, comment, or message to be posted.

AI Post-Moderation

Our AI post-moderation feature goes beyond the basics, offering an enhanced moderation experience. It scans a wider range of content types across more moderation categories and provides more flexibility in configuring actions to be taken based on flag and block confidence levels.

Our AI post-moderation feature supports moderating text, image, and video content in messages; text, image, video, and livestream content in posts; and text and image content in comments.

AI Text Post-Moderation

Our AI text post-moderation feature detects and moderates text on posts, comments, and messages that are:

  • Sexually explicit or adult in certain situations

  • Sexually suggestive or mature in certain situations

  • Offensive in certain situations

AI Image & Video Post-Moderation

Our AI image and video post-moderation feature detects images and videos on posts and messages, and images in comments that contain imagery depicting:

  • Adult Toys

  • Air crash

  • Alcohol

  • Alcoholic Beverages

  • Bare-chested Male

  • Corpses

  • Drinking

  • Drug Paraphernalia

  • Drug Products

  • Drug Use

  • Drugs

  • Emaciated Bodies

  • Explicit Nudity

  • Explosions and blasts

  • Extremist

  • Female Swimwear Or Underwear

  • Gambling

  • Graphic Female Nudity

  • Graphic Male Nudity

  • Graphic Violence Or Gore

  • Hanging

  • Hate Symbols

  • Illustrated Explicit Nudity

  • Male Swimwear Or Underwear

  • Middle Finger

  • Nazi Party

  • Nudity

  • Partial Nudity

  • Physical Violence

  • Pills

  • Revealing Clothes

  • Rude Gestures

  • Self Injury

  • Sexual Activity

  • Sexual Situations

  • Smoking

  • Suggestive

  • Tobacco

  • Tobacco Products

  • Violence

  • Visually Disturbing

  • Weapon Violence

  • Weapons

  • White Supremacy

How It Works

Moderation is performed using two main factors - flagConfidence and blockConfidence. When a post, comment, or message is successfully created, our AI system will automatically scan both the text and media, generating a confidence value as a result. If this confidence value falls below flagConfidence, the post, comment, and message pass moderation. In cases where the confidence value falls between flagConfidence and blockConfidence, our moderation feature will flag the content for review.

You will be able to review the flagged content in Social Plus Console or listen to Social Plus's real-time flagged post, flagged comment, or flagged message event through your webhook on the server side - allowing you to implement appropriate actions to be taken based on your moderation policies. Finally, if the confidence value surpasses blockConfidence, the system deletes the content altogether, ensuring a responsive approach to maintaining a safe online environment.

As a default configuration, all categories have an initial flagConfidence value of 40, and an initial blockConfidencevalue of 80.

Post-Moderation Description

Name
Data Type
Description

category

String

Name of each moderation's category

flagConfidence

Number

Value of moderation category’s flag confidence

blockConfidence

Number

Value of moderation category’s block confidence

moderationType

String

Value of moderation type for the category. There are 2 possible values, "text" or "media"

Specify API Endpoint

To use the enhanced moderation APIs, specify the appropriate API endpoint on each HTTP request. Each data center has a unique endpoint, so it's essential to adjust it accordingly. By selecting the correct endpoint associated with the right location, you ensure faster response times and optimize the overall performance of your API requests.

Region
API Endpoint

Europe

Singapore

United States

Get Moderation Confidence Level

Get moderation confidence level API can be used to retrieve the confidence level for each moderation category. The API will return a list of moderation categories along with the corresponding confidence level.

Update Moderation Confidence Level

Set moderation confidence level API can be used to set confidence level to any moderation category. The API will return list of moderation categories along with the corresponding confidence level.

We recommend adjusting the confidence levels for each category based on your moderation policies and the needs of your community. By doing so, you can ensure that Enhance Moderation provides the optimal level of moderation for your platform.

https://beta-eu.amity.services/
https://beta-sg.amity.services/
https://beta-us.amity.services/
support team

Retrieve confidence level for each moderation category

get
Authorizations
Header parameters
AuthorizationstringRequired
Responses
200
Moderation categories and confidence level JSON
application/json
get
GET /v1/content-moderation/confidences HTTP/1.1
Host: API_ENDPOINT{basePath}
Authorization: text
Accept: */*
200

Moderation categories and confidence level JSON

{
  "confidences": [
    {
      "category": "Alcohol",
      "flagConfidence": 40,
      "blockConfidence": 80,
      "moderationType": "media"
    },
    {
      "category": "Drugs",
      "flagConfidence": 30,
      "blockConfidence": 80,
      "moderationType": "media"
    }
  ]
}
  • Enabling AI Content Moderation
  • AI Pre-Moderation
  • AI Post-Moderation
  • AI Text Post-Moderation
  • AI Image & Video Post-Moderation
  • How It Works
  • Post-Moderation Description
  • Specify API Endpoint
  • Get Moderation Confidence Level
  • GETRetrieve confidence level for each moderation category
  • Update Moderation Confidence Level
  • PUTSet confidence level to the moderation category

Set confidence level to the moderation category

put
Authorizations
Header parameters
AuthorizationstringRequired
Body
Responses
200
Request has succeeded
application/json
put
PUT /v1/content-moderation/confidences HTTP/1.1
Host: API_ENDPOINT{basePath}
Authorization: text
Content-Type: application/json
Accept: */*
Content-Length: 143

{
  "confidences": [
    {
      "category": "Alcohol",
      "flagConfidence": 40,
      "blockConfidence": 80
    },
    {
      "category": "Drugs",
      "flagConfidence": 30,
      "blockConfidence": 80
    }
  ]
}
200

Request has succeeded

{
  "data": [
    {
      "message": "Success"
    }
  ]
}