Story clustering

The duplication problem is worse than it looks

News is heavily syndicated. A wire report gets picked up by hundreds of sites, lightly re-headlined, and published within minutes of each other. A search that ranks articles will faithfully return all of them, because by any reasonable relevance measure they are all relevant.

For a human reading a feed this is mild noise; they skim past it. For a model it is actively misleading in two separate ways. First, the context window fills with one fact repeated, so fewer distinct facts fit and the answer gets narrower. Second — and this is the part people miss — repetition reads as significance. A model shown the same claim forty times will weight it as well-corroborated, when in reality it has seen one source forty times. Syndication volume tracks distribution deals, not truth.

What a clustered result gives you

Each story arrives as one item with a count and the outlets that ran it. That outlet list is usually the most informative field in the response, because it separates two situations that raw article volume conflates: a story carried by three publishers repeatedly, and a story carried once each by thirty independent ones. The second is a real event with broad attention. The first is one newsroom with a content schedule. Outlets are counted as publisher domains, and a wire service, an ownership group or a broadcast-affiliate network puts one newsroom's report on many of them, so a wide outlet list is breadth of distribution first and independent reporting second. The list itself is what lets you tell the two apart. sort=outlets orders stories by that breadth instead of by article count, and sort=recency puts the newest first.

This makes escalation rules simple enough to write in a few lines and reason about later. One outlet publishing is a mention. Five unrelated outlets inside an hour is a story. A story that keeps acquiring outlets after the first hour is the one that ends up in front of an executive. Those are counts over data you already have, not a model you have to train, evaluate and maintain.

When you want the duplicates

Clustering is not always right, and it is a separate call rather than a global setting for that reason. If you are studying how coverage propagates — which outlet ran it first, how the framing shifted between the wire copy and the regional rewrite, which markets picked it up at all — the duplicates are your data and collapsing them destroys the thing you are measuring.

The rule of thumb: cluster when results go into a context window or an alert, and do not cluster when results are the object of study. Media analysis, propagation research and framing comparison all want the raw articles.

An expanded story sits between the two. It carries first_reported, the earliest of its articles the index holds, and title_variants, the distinct headlines among them. They are proxies for who had it first and how many separate write-ups exist, not lineage. Nothing in the index marks wire copy, and the earliest article indexed is not always the first one published.

Following a story over time

Because a story keeps its identity as coverage accumulates, you can check for what is new without re-reading what you have already seen, and you can deduplicate on the story rather than on the headline. That distinction matters more than it sounds: headlines get rewritten between editions, so headline-based deduplication silently re-admits the same story every time an editor changes a word, and the resulting duplicate alerts are the usual reason a monitoring channel stops being read.

A story can also be fetched by its identifier: get_story over MCP, GET /stories/{id} over REST. One call returns the whole cluster — the article and outlet counts, every outlet it has reached so far, and the first and last time it was seen — without re-running the search that found it. That is the number an escalation rule should read. Growth inside one polling window says how much arrived since the last poll; growth measured against the whole story says whether a mention has become an event, and only the second survives a missed poll.

A watch can do the checking. It runs a story query on an interval and records an event when a story first reaches the number of outlets you set, and again as it crosses 5, 15, 40 and 100 outlets, so growth arrives as an event instead of a loop you maintain. The MCP page covers watches, their webhooks and what each run costs.

Reference

Every parameter, what it accepts and how it behaves against the index: the /stories reference. What one expanded story carries, field by field: the /stories/{id} reference.