UnzoiBot
You probably found this in your server logs. UnzoiBot fetches news articles so they can be found by search. It obeys robots.txt, crawls slowly, and never republishes your article text.
What is UnzoiBot?
UnzoiBot is the crawler for Unzoi, a news search API. It fetches news article pages so they can be indexed for search and retrieval. It respects robots.txt and rate-limits itself to 20 requests per minute per domain — one request every three seconds, per host, regardless of how many other sites are being fetched at the same moment. It is not an AI training crawler: what it collects builds a retrieval index whose results point back at your pages.
How do I block UnzoiBot?
Add this to your robots.txt:
User-agent: UnzoiBot
Disallow: /
We re-read robots.txt at least once a day, so a block takes effect within 24 hours.
To restrict rather than block, ordinary Disallow and Allow
rules work as expected:
User-agent: UnzoiBot
Disallow: /archive/
Crawl-delay: 10
A Crawl-delay slower than our default is honoured. A faster one is
ignored — we will not crawl you harder than 20 requests per minute whatever your
robots.txt permits.
How it behaves
| User-agent token | UnzoiBot |
|---|---|
| Full user-agent string | UnzoiBot/1.0 (+https://www.unzoi.com/crawler) |
| Requests per minute, per domain | 20 (one every 3 seconds) |
| Crawl-delay | Honoured, and used as a floor if slower than our default |
| Request timeout | 20 seconds |
| Maximum response size | 4 MB |
| Content types fetched | HTML only |
| Retries | Up to 3, with backoff on 429 and 5xx |
| JavaScript execution | Off by default |
| Forms, logins, paywalled content | Never |
Rate limiting is per domain, not per crawler. Sixteen fetches may be in flight across the whole internet at once, but only one of them is ever yours, and the next one waits three seconds.
Does UnzoiBot republish article text?
No. Search results return a headline, a short snippet, the publication name, a timestamp and a link to the original article, and full article text is never served to API users. The distinction matters commercially rather than only technically: a result that carries your headline and a link sends a reader to you, whereas a result carrying the article does not, and the second is what publishers are objecting to when they block crawlers.
What we do with the page
We extract the article text, the headline, the publication date and the author line where one is present. The text is used to build a search index. What an API user receives is a headline, a short snippet, your publication name, a timestamp and a link back to you.
Article bodies are never served to API users and never sold. Every result carries the link, so a reader who wants the article visits your site to read it.
What it does not do
- Submit forms, log in, or attempt to reach anything behind authentication
- Bypass paywalls, metering or consent walls
- Fetch images, video, stylesheets or scripts — HTML only
- Crawl a domain faster than 20 requests per minute, regardless of robots.txt
- Ignore a
Disallowrule, a429, or a503
Verifying it is us
User-agent strings are trivially forged, and a bot claiming to be UnzoiBot may not be.
To confirm, run a reverse DNS lookup on the requesting IP and check that it resolves
to a hostname under crawl.unzoi.com, then forward-resolve that hostname
and confirm it matches the original IP.
$ dig -x 203.0.113.42 +short
crawl-203-0-113-42.crawl.unzoi.com.
$ dig +short crawl-203-0-113-42.crawl.unzoi.com
203.0.113.42Anything failing that check is not us, and we would like to know about it. Traffic that fails verification can be blocked without affecting your presence in our index.
If we are causing problems
Tell us and we will stop. The contact form reaches a person, and crawler reports are treated as urgent. Include your domain and a rough timestamp and we can identify the traffic.
We will also reduce our rate on request, without you needing to block us. If the default pace is too much for your infrastructure, ask and we will slow down.