Home / Recipes / Stop one wire report arriving as forty results
Worked queryStop one wire report arriving as forty results
/stories in hybrid mode. The parameters matter less
than the reasoning behind them, so that is most of what is below.
The request
GET /stories
?q=<topic>
&mode=hybrid
&from=2026-07-29Why these parameters
Syndication is structural, not incidental. A wire report gets picked up by hundreds of sites, lightly re-headlined, within minutes. Every one of those is relevant by any ranking measure, so no amount of relevance tuning removes them — the fix has to happen at the response shape. Collapsing by story does that, and it is why the endpoint differs rather than a parameter.
The obvious approach, and why it loses
The obvious approach is to deduplicate in your own code by comparing titles, and it works well enough on the easy cases to convince people it works. Wire pickups are not the easy case: outlets re-headline the same copy, add a local angle to the first paragraph, and publish regional editions that differ by a place name. Title similarity either merges genuinely distinct regional stories or leaves near-identical ones separate, and which it does depends on a threshold that has no correct value. The second instinct, deduplicating by URL or by publisher, does nothing at all, because syndication is precisely the case where the publisher differs and the content does not.
When it returns too much, or too little
There is no threshold to tune here, which is the point; what you tune is the window. Too much means the window is long enough that several distinct developments in the same subject have accumulated, and the result is a list of stories rather than a duplicate problem. Too little usually means the window is short enough to catch a story mid-propagation, where the cluster has formed around the first few pickups and later ones have not arrived. If you need to see inside a cluster — to check what was merged, or to count actual articles — query the members through search rather than loosening the collapse, which is the control the endpoint choice was making for you.
How you would know it is working
Take a day you can inspect and count by hand. Pick one event you know was heavily syndicated, retrieve both ways over the same window, and compare: the article count is what your users would have seen, and the story count is what they should have. The ratio between them on a busy day is the best single argument for the change, and it is usually larger than anyone expects. Then check the other direction on a handful of clusters — read the members and look for anything merged that should not have been. Both numbers are worth recording once, because they are the evidence you will be asked for later.
When this is the wrong tool
Story collapsing is a judgement about what constitutes one event, and judgements can be wrong at the margins. Two distinct but closely related developments occasionally collapse together, and a story that evolves substantially over days may stay under one identifier when you would rather it split. If your analysis depends on exact event boundaries — legal, compliance or academic work — verify the clusters rather than trusting them, and use get_story or /search to inspect the members of any cluster that matters. This is also the wrong recipe when the duplicates are load-bearing: propagation studies, framing comparison and anything measuring reach all need the raw articles, and collapsing them destroys the measurement rather than cleaning it up.
Running it for real
Response fields, failure modes, adaptations and scheduling cost: Stop one wire report arriving as forty results, in the docs.