Home / Recipes / Follow one story as coverage accumulates, without re-reading it
Worked queryFollow one story as coverage accumulates, without re-reading it
/stories in hybrid mode. The parameters matter less
than the reasoning behind them, so that is most of what is below.
The request
GET /stories
?q=<topic>
&mode=hybrid
&from=<last poll timestamp>
# then, for the story you care about:
GET /search?story_id=<id>&from=<last poll>
# the whole story at once: counts, every outlet, first and last seen
GET /stories/<id>
# or let the index poll: new and growing stories become events
POST /watches {"q": "<topic>", "mode": "hybrid", "interval_minutes": 60}
GET /watches/<watch id>/eventsWhy these parameters
Poll for what is new rather than re-querying the whole window, and keep the last-seen timestamp yourself, or save the topic query as a watch, which polls and keeps that position server-side. Once you have identified the story that matters, switching to its identifier is both cheaper and more precise than repeating a text query that may drift onto adjacent events as coverage evolves. For the whole-story counts — every outlet, first and last seen — fetch the identifier with get_story rather than summing polls.
The obvious approach, and why it loses
The obvious approach is to re-run the original text query on a schedule and diff the results, which works for a day or two and then drifts. As a story develops, the vocabulary that describes it changes: the query written when an incident was first reported stops matching once coverage moves to the response, the inquiry and the consequences. You end up either widening the query, which pulls in adjacent events, or keeping it narrow and watching the feed go quiet while the story continues. Switching to the story identifier fixes the drift in the opposite direction — the cluster follows the event rather than the words.
When it returns too much, or too little
Too much means the cluster has absorbed a neighbouring event, which happens when two related developments are covered in the same articles. Inspect the members with get_story, or a search on the identifier, before assuming the extra volume is your story. Too little, after a period of normal flow, is the characteristic failure of this recipe and it does not announce itself: a long-running situation eventually spawns a new cluster and the old identifier keeps returning nothing, which looks exactly like the story ending. Run the topic query alongside the identifier poll on a slower cadence and reconcile the two, so a silent split shows up as a discrepancy rather than as silence.
How you would know it is working
The metric is whether the tracker and a person reading the coverage would describe the same story. Once a week, run the topic query over the same period and compare its clusters against what the identifier returned. A tracker in good health accounts for most of the coverage the topic query finds; one that has drifted accounts for very little, and the gap appears before anyone notices the feed has gone quiet. Also record how long you have been following a single identifier. Past a few weeks on an evolving situation, treat continuity as a claim that needs re-checking rather than a property of the system.
When this is the wrong tool
Story identifiers are stable but not permanent guarantees. A long-running situation will eventually produce new clusters as its subject shifts, so a tracker pinned to one identifier will slowly stop seeing the thing it was following without ever reporting an error. Re-run the topic query periodically alongside the identifier poll and reconcile, rather than trusting the identifier indefinitely. This recipe is also unsuited to measuring how a story was covered rather than what happened next — for that you want the full article set, not the increment, and the polling pattern here deliberately discards everything you have already seen.
Running it for real
Response fields, failure modes, adaptations and scheduling cost: Follow one story as coverage accumulates, without re-reading it, in the docs.