Someone joins the same game every evening. At first, they talk constantly. They joke, ask questions, and invite others to play. Then their messages get shorter. A week later, they stop using the chat. Soon after, they stop showing up.
No report was filed. No single message broke a rule.
But something changed.
If no one hit the report button, was anything wrong?
Reports show only what people manage to report
Reporting systems are necessary. They are also one of the narrowest windows a platform has into its own community. Before a report exists, someone has to notice the behavior, decide it crossed a line, believe the line matters, trust that reporting will help, and find the energy to act.
At every step, some people stop.
They may be unsure whether it was serious enough. Children and teenagers may lack the words to explain what happened, or the confidence to see that it was not their fault. Some worry that reporting will make things worse, especially if the other person could identify them. Others have already decided the community is not worth the effort.
A report marks the moment the system finds out about harm. The harm itself usually started earlier.
Harm often shows up as a pattern
One questionable message, read on its own, can look harmless. A series of messages, read in context, can tell a different story. The signals worth noticing are rarely dramatic:
- The same person testing boundaries, going a little further each time
- One user following another between rooms, lobbies, or conversations
- Jokes becoming more personal and targeted with each repetition
- People going quiet or leaving when a particular account appears
- Several small incidents pointing toward the same underlying risk
A drop in participation tells you where to look. Finding out what happened takes more.
These signals live between messages, out of reach of any banned words list: who said what, to whom, how often, after which event, and what happened next. Moderation that reads only individual messages will miss the pattern that made those messages harmful.
Technology widens the view
This is where software helps, in a more modest role than most sales decks suggest (ours included). Software handles scale and repetition well. It can process more activity than any human team, follow behavior across sessions, and connect events a moderator would never see side by side. That lets it flag emerging risks, handle routine actions, organize reports into cases, and put the most urgent situations at the top of the queue. [Link: moderation architecture post]
Understanding what a community means to the people in it is a different job. Two friends who tease each other every day and one person slowly wearing another down can look almost identical in the data. Telling them apart takes context, history, and a human who knows the community.
Behavioral signals indicate risk. Confirming harm takes human judgment.
Safety means measuring more than removal
Most safety metrics count enforcement: content removed, accounts restricted, response times. These numbers show how well a platform responds to visible problems. They say little about whether people feel able to take part, or whether they want to stay.
The harder questions are about participation. Do new members speak in their first week, or wait to see what happens to those who do? Can someone ask an obvious question without bracing for ridicule? How much does a person have to absorb before anyone notices? And when they finally speak up, does anything change?
Some of this can be tracked: how many new members post in their first week, whether people go quiet after contact with specific accounts, and whether people who report hear back.
A community where people talk freely has achieved something no takedown count can show.
Regulation is moving earlier too
Regulators and industry analysts are reaching the same conclusion. The European Commission’s guidelines on protecting minors under the Digital Services Act, published on 14 July 2025, put heavy weight on how platforms are designed. They recommend letting children block and mute any user, preventing them from being added to groups without consent, and improving reporting tools so users get prompt feedback. The guidelines are voluntary, but the Commission uses them as a benchmark for compliance with Article 28.
PwC’s Trust and Safety Outlook 2026 describes a similar shift inside organizations. It argues that Trust and Safety needs to move from a reactive operating function to a strategic capability built into product design, governance, and operations. Trust, it says, should be designed earlier, measured more precisely, and governed continuously.
Safety work that starts only when a report arrives has started too late.
