Story clustering

The duplication problem is worse than it looks

News is heavily syndicated. A wire report gets picked up by hundreds of sites, lightly re-headlined, and published within minutes of each other. A search that ranks articles will faithfully return all of them, because by any reasonable relevance measure they are all relevant.

For a human reading a feed this is mild noise; they skim past it. For a model it is actively misleading in two separate ways. First, the context window fills with one fact repeated, so fewer distinct facts fit and the answer gets narrower. Second — and this is the part people miss — repetition reads as significance. A model shown the same claim forty times will weight it as well-corroborated, when in reality it has seen one source forty times. Syndication volume tracks distribution deals, not truth.

What a clustered result gives you

Each story arrives as one item with a count and the outlets that ran it. That outlet list is usually the most informative field in the response, because it separates two situations that raw article volume conflates: a story carried by three publishers repeatedly, and a story carried once each by thirty independent ones. The second is a real event with broad attention. The first is one newsroom with a content schedule.

This makes escalation rules simple enough to write in a few lines and reason about later. One outlet publishing is a mention. Five unrelated outlets inside an hour is a story. A story that keeps acquiring outlets after the first hour is the one that ends up in front of an executive. Those are counts over data you already have, not a model you have to train, evaluate and maintain.

When you want the duplicates

Clustering is not always right, and it is a separate call rather than a global setting for that reason. If you are studying how coverage propagates — which outlet ran it first, how the framing shifted between the wire copy and the regional rewrite, which markets picked it up at all — the duplicates are your data and collapsing them destroys the thing you are measuring.

The rule of thumb: cluster when results go into a context window or an alert, and do not cluster when results are the object of study. Media analysis, propagation research and framing comparison all want the raw articles.

Following a story over time

Because a story keeps its identity as coverage accumulates, you can poll for what is new without re-reading what you have already seen, and you can deduplicate on the story rather than on the headline. That distinction matters more than it sounds: headlines get rewritten between editions, so headline-based deduplication silently re-admits the same story every time an editor changes a word, and the resulting duplicate alerts are the usual reason a monitoring channel stops being read.