Language coverage

English-first design has a visible cliff

Most news APIs were built for English and extended outward. You can usually see where the extension stopped: filters exist for many languages, but the retrieval quality behind them degrades sharply, because the tokenizer, the ranking and the evaluation set were all tuned on English text.

The failure is quiet. A query in Thai or Hindi returns results, they look plausible, and nothing indicates that most of the relevant coverage was never a candidate. Teams typically discover this when a major story in a market they cover fails to trigger anything, and then cannot tell whether the problem is their query, their filter or the index.

Why script determines which retrieval mode to use

Keyword matching depends on assumptions that hold in English and fail elsewhere, so the right mode is a property of the language rather than a preference.

The short version: prefer semantic or hybrid mode outside English, and reserve keyword mode for identifiers that publishers render consistently, such as tickers.

What a language count actually tells you

Advertised language counts across this category vary by an order of magnitude for corpora that are probably comparable, and no vendor publishes how the number is computed. A language with two outlets and a language with two hundred both increment the count by one.

So we publish the distribution instead. Per-language pages carry article volume, distinct outlet count and observed history, regenerated from the index each publication cycle. Languages without enough outlets to answer a real question get no page, rather than a page implying coverage that is not there.

Outlet spread is the number to watch. Volume tells you how much was published; spread tells you how many independent voices you are actually reaching. A language served by three prolific publishers will out-produce one served by fifty modest ones and give you a far narrower picture.

Languages with a coverage page

A language appears here when the index carries enough of it, from enough distinct outlets, to answer a real question — and when we have something to say about querying it beyond the numbers. Languages that clear only one of those two bars deliberately have no page.

Figures are provisional — measured from the public source the index is built from rather than from the index itself, pending a snapshot against the API.