UnzoiBot is the crawler for Unzoi, a news search API. It fetches news article pages so they can be indexed for search and retrieval. It respects robots.txt and rate-limits itself to 20 requests per minute per domain.

How do I block UnzoiBot?

Add a Disallow rule for user-agent UnzoiBot in your robots.txt. The change takes effect within 24 hours, when the cached copy expires.

Does UnzoiBot republish article text?

No. Search results return a headline, a short snippet, the publication name, a timestamp and a link to the original article. Full article text is never served to API users.

For publishers

UnzoiBot

You probably found this in your server logs. UnzoiBot fetches news articles so they can be found by search. It obeys robots.txt, crawls slowly, and never republishes your article text.

Blocking it

Add this to your robots.txt:

User-agent: UnzoiBot
Disallow: /

We re-read robots.txt at least once a day, so a block takes effect within 24 hours. To restrict rather than block, ordinary Disallow and Allow rules work as expected:

User-agent: UnzoiBot
Disallow: /archive/
Crawl-delay: 10

A Crawl-delay slower than our default is honoured. A faster one is ignored — we will not crawl you harder than 20 requests per minute whatever your robots.txt permits.

How it behaves

User-agent token	UnzoiBot
Full user-agent string	UnzoiBot/1.0 (+https://www.unzoi.com/crawler)
Requests per minute, per domain	20 (one every 3 seconds)
Crawl-delay	Honoured, and used as a floor if slower than our default
Request timeout	20 seconds
Maximum response size	4 MB
Content types fetched	HTML only
Retries	Up to 3, with backoff on 429 and 5xx
JavaScript execution	Off by default
Forms, logins, paywalled content	Never

Rate limiting is per domain, not per crawler. Sixteen fetches may be in flight across the whole internet at once, but only one of them is ever yours, and the next one waits three seconds.

Every fetch passes the robots check and the per-domain limiter first.

What we do with the page

We extract the article text, the headline, the publication date and the author line where one is present. The text is used to build a search index. What an API user receives is a headline, a short snippet, your publication name, a timestamp and a link back to you.

Article bodies are never served to API users and never sold. Every result carries the link, so a reader who wants the article visits your site to read it.

What it does not do

Submit forms, log in, or attempt to reach anything behind authentication
Bypass paywalls, metering or consent walls
Fetch images, video, stylesheets or scripts — HTML only
Crawl a domain faster than 20 requests per minute, regardless of robots.txt
Ignore a Disallow rule, a 429, or a 503

Verifying it is us

User-agent strings are trivially forged, and a bot claiming to be UnzoiBot may not be. To confirm, run a reverse DNS lookup on the requesting IP and check that it resolves to a hostname under crawl.unzoi.com, then forward-resolve that hostname and confirm it matches the original IP.

$ dig -x 203.0.113.42 +short
crawl-203-0-113-42.crawl.unzoi.com.

$ dig +short crawl-203-0-113-42.crawl.unzoi.com
203.0.113.42

Anything failing that check is not us, and we would like to know about it. Traffic that fails verification can be blocked without affecting your presence in our index.

If we are causing problems

Tell us and we will stop. The contact form reaches a person, and crawler reports are treated as urgent. Include your domain and a rough timestamp and we can identify the traffic.

We will also reduce our rate on request, without you needing to block us. If the default pace is too much for your infrastructure, ask and we will slow down.