About fifty thousand questions, answers, and comments are published on gutefrage every day. 10% of published items get deleted because they violate gutefrage quality requirements: An item can be of trolling or offensive nature, it can contain forbidden sensitive content or dangerous and illegal information. There are two ways to capture “bad” items: A user can submit a complaint, or it can be stumbled upon by a moderator and then be deleted manually. This approach has two major drawbacks. First, it is time-consuming and cost-inefficient for moderators to rely on luck to find bad content. Second, it negatively affects user experience and creates “broken windows theory” thus causing higher user churn-rate and increasing amount of low-quality content.

To address these problems, we have developed pre-moderation – a system of ML-based classifiers that automatically detects suspicious content based on user characteristics and past behavior combined with content-based features and semantic checks. Depending on classifier confidence, a suspicious item can be either deleted automatically or locked and sent for manual check. The system is continuously monitored and optimized by adjusting model thresholds and retraining. Pre-moderation has enabled us to cost-effectively organize the workload of moderators and to reduce bad content reporting rate.