How Facebook prioritizes content for review

UPDATED

JUL 29, 2021

Facebook’s technology detects and removes the majority of violating content before it’s ever reported. When someone posts on the Facebook app or Instagram, our technology checks to see if the content goes against the Facebook Community Standards and Instagram Community Guidelines. In most cases, identification is a simple matter. The post either clearly violates our policies or it doesn’t.

Other times, identification is more difficult. Perhaps the sentiment of the post is unclear, its language is particularly complex or its imagery too context-dependent. In these cases, we conduct further review using people.

Prioritizing factors for human review

When determining which content our human review teams should review first, we consider 3 main factors:

SEVERITY

How likely is it that the content could lead to harm, both online and offline?

VIRALITY

How quickly is the content being shared?

LIKELIHOOD OF VIOLATING

How likely is it that the content in question does in fact violate our policies?

Because we want to prevent as much harm as possible, our review systems use technology to prioritize high-severity content with the potential for offline harm and viral content which is spreading quickly.

Human review teams help our technology improve

Our human review teams use their expertise in certain policy areas and locales to make difficult, often nuanced judgment calls. Every time reviewers make a decision, we use that information to train our technology. Over time, across millions of decisions, our technology gets better, allowing us to remove more violating content.

Helping reviewers make the right calls

Technology helps human review teams do what they do best

Like many machine learning models, our technology improves over time as it receives more examples of violating content. This means human review teams have been able to focus more on severe, viral, nuanced, novel and complex content—exactly the sort of decisions where people tend to make better decisions than technology.

Common questions about content review prioritization

How has enforcement prioritization changed over time?

Previously, human review teams would spend the vast majority of their time reviewing content reported by people. This meant they were often spending too much time on low-severity or clearly non-violating content and not enough time on the severest content with the greatest potential for harm. It also meant many human decisions weren’t that useful for improving our enforcement technology.

Our current approach to prioritization addresses these issues, allowing us to review the most potentially harmful content first and improve our technology faster.

Do human review teams review every user report?

Not necessarily. Both human review teams and technology play a role in reviewing user reports. In cases where our technology can analyze a given piece of content, it will automatically take action—or not—on the content in question.

How do you ensure that human decisions don’t introduce bias into your technology?

To address fairness and inclusion concerns associated with the deployment of AI in Facebook technologies, we created our Responsible AI team—a dedicated, multidisciplinary team of ethicists, social and political scientists, policy experts, artificial intelligence researchers and engineers. The team’s overall goal is to develop guidelines, tools and processes to tackle issues of AI responsibility and help ensure these systemic resources are widely available across Facebook.