Home / How it works / Evaluation
Measurements, defined before they are publishedEvaluation
We have not yet published a measurement of how well Unzoi clusters, retrieves or resolves. This page says what each measurement will be, precisely enough to implement, and which part of the roadmap cannot ship until its result is public.
What every answer states about itself today
Quality figures are not published yet. What is published is the answer's own account of its limits, as fields a model can read rather than infer. These are live on the REST API and on all 20 MCP tools, and they are the reason an agent built on Unzoi can say "I do not know" instead of confidently reporting a gap as an absence.
| Field | What it tells the caller |
|---|---|
partial, coverage | Whether every part of the index answered, and if not, why — a field not yet built there, a part that timed out, a window before the archive begins. |
from_clamped | That the window you asked for was cut to the archive depth your plan includes. |
approximate counts | Which totals are estimates rather than exact counts. |
estimate, max_credits | What a call will cost before it runs, and a ceiling it will refuse to exceed. |
entity_resolution_version | The rule set that grouped the spellings, with a confidence on each match. |
events-v1 | The versioned definition of the event classes, named in the published schema, so a change of definition is a new version rather than a silent shift. |
note, indexed_through | On relationship answers: that co-mention is association and not partnership, ownership or cause, and how recent the evidence is. |
What will be measured
One section per capability. Each names the question a caller would ask, the metrics that answer it, and what waits on the result. The status is of the measurement, not of the capability: story clustering is live, and its measurement is planned.
Story clustering Planned
When the index says two articles are one event, is it right, and does it miss any?
- Pairwise precision and recall
- Over every pair of articles in a labelled sample, the share of pairs placed in one story that belong together, and the share that belong together and were placed together.
- B-Cubed F1
- Per-article precision and recall of its story against the labelled event, averaged, so one large story cannot dominate the score.
- Fragmentation rate
- Labelled events that were split across two or more stories, as a share of all labelled events.
- Accidental-merge rate
- Stories that contain articles from two or more labelled events, as a share of all stories.
- Stability
- How often an article changes story between two consecutive rebuilds of the same window.
What waits on it. Cross-lingual event identity (roadmap item 5) may not ship until story-match precision and the false-merge rate are published for at least one language pair.
Retrieval Planned
Does the right article come back near the top, and does that hold for each mode?
- Recall@10
- The share of labelled relevant articles that appear in the first ten results, per query, averaged.
- nDCG@10
- Graded relevance of the first ten results, discounted by rank and normalised against the best possible ordering.
- MRR
- The reciprocal rank of the first relevant result, averaged over queries.
- Per mode
- Each of the above reported separately for keyword, semantic and hybrid, over the same queries and filters.
What waits on it. No roadmap item waits on it. It is meant as a regression gate: a change to the embedding model or the index format would not ship if these fell.
An internal benchmark checks that articles from the same story retrieve one another (recall@10) when the embedding model or vector compression changes. Its results are not published, and same-story recall is not relevance.
Multilingual retrieval Planned
Is retrieval in Marathi or Arabic as good as in English, and does an English query reach them?
- Per language
- The retrieval metrics above, reported for each language with enough labelled queries, never averaged into one figure.
- Per language pair
- Recall when the query is in one language and the relevant articles are in another.
What waits on it. Part of the cross-lingual gate (roadmap item 5). Indian languages and Arabic are the first targets.
Enrichment Planned
Are the entities, places, amounts and signals on an article actually in it?
- Precision and recall per axis
- For each field — people, organisations, places, dates, amounts, each signal — against a labelled sample of articles.
- Per language
- The same, for every language the fields are served in.
What waits on it. Native enrichment (roadmap item 2) serves no field until its precision and recall are published per axis and per language.
Entity resolution Planned
When a name resolves to an id, are the spellings grouped under it the same entity?
- Alias precision and recall
- Of the spellings grouped under an id, the share that name that entity; of the spellings the index holds for it, the share that were grouped.
- Collision rate
- Ids that group spellings of two or more distinct entities, as a share of ids in the labelled set.
- Ambiguity set
- A fixed list of names shared by different entities, and whether each resolves to separate ids.
What waits on it. No roadmap item waits on it. Resolution is rule-based normalisation (er-v1) with a confidence per match, and the version is on every answer, so a change of rules is visible.
Relationship evidence Planned
When the graph says two entities are reported together, do the evidence articles show it?
- Evidence precision
- The share of returned evidence articles that do mention both entities, judged by a reviewer.
- Ranking agreement
- How closely the order of related entities matches a reviewed set for the same seed and window.
What waits on it. The graph projection (GRAPH_ROADMAP phase 1) starts by defining relevance and evidence-quality evaluation sets, and is compared against them before it replaces the query-time graph.
Syndication and first report Planned
When the index says an outlet broke a story, did it?
- Wire-pickup error rate
- Against a hand-labelled set of known wire pickups, the share where the originating report or the carrying outlets are identified wrongly.
- First-report accuracy
- The share of labelled stories whose first reporting outlet the index names correctly.
What waits on it. Publisher profiles (roadmap item 3) stay "being built" until this error rate is published beside the metric.
Watches Planned
How late, and how noisy, is an alert?
- Detection delay
- Time from an event’s first indexed article to the watch event that reports it.
- Duplicate-alert rate
- Watch events that report a story an earlier event already reported, as a share of all events.
- False escalation rate
- Events marking a story as growing when a reviewer judges it has not.
What waits on it. Change detection (roadmap item 7) extends watches to first reports and cross-border spread; it would be measured the same way.
Tool selection Planned
Given the MCP tool list and a request in plain language, does a model call the right tool with the right arguments?
- Tool accuracy
- Over a fixed set of natural-language intents, each labelled with the expected tool, the share where the model chose it.
- Parameter accuracy
- Of those, the share whose arguments match the labelled ones.
- Calls and credits to answer
- How many tool calls, and how many credits, a model spends before it has what the intent needs.
- Per model
- Reported separately for each widely used model family, since tool choice varies by model rather than by server.
What waits on it. No roadmap item waits on it. It is how the tool list is judged: whether overlapping tools confuse models, and whether a tool should be merged or split.
Freshness and latency Planned
How long from publication to searchable, and how long does a call take?
- Ingest latency, p50 and p95
- Time from an article’s publication timestamp to its first appearance in query results.
- Query latency, p50 and p95
- Server time per call, by operation and retrieval mode.
- Index lag
- How far behind the newest article in the index is, as graph answers already report in indexed_through.
What waits on it. No roadmap item waits on it. It is what a service-level objective would be written against.
A probe that asks the public API every minute for what became searchable since its last call is built and not yet running. Once it runs, its last 24 hours of figures, from publication and from collection to searchable with the number of articles behind each, appear on the live demo page. Request latency and ingestion backlog are recorded internally for operating the service. No figures are published yet, and none should be inferred from this page.
The rules a published number will follow
- The labelled set is described: how it was sampled, how many items, which languages, who labelled it and how disagreements were settled.
- Results are reported per language and per retrieval mode, never averaged into one headline figure that hides the weakest language.
- The error rate is published beside the metric, and a capability gated on a measurement stays "being built" until both are public.
- Every metric is re-run on every change to a model, a rule set or the index format, and a regression blocks the change rather than being footnoted after it.
- A versioned rule set (entity resolution, event classes, signals) is measured per version, so a result always names what it measured.
Evaluating it yourself, in the meantime
Until these are published, your own queries are the only evidence that settles anything, and they are more relevant to you than ours would be. The comparison guide sets out the tests that separate news APIs in twenty minutes, and every recipe ends with how to check whether it is working on your data. The free plan includes every retrieval mode, so the test costs nothing but the queries.
Why not publish a number now?
Because a figure without its labelled set, its per-language breakdown and its error rate is not something a reader can check, and an averaged figure hides exactly the languages and entities where retrieval is weakest. The definitions come first so the numbers, when they arrive, can be held to them.
Roadmap gates quoted as of 2026-09-20.