UnlobBot
The crawler that builds our index. This page documents exactly how it behaves and how to control it — because a crawler that cannot be controlled has no business on your site.
How to identify us
One user-agent string, on every request, including the fetch of your robots.txt. We do not rotate it, shorten it under load, or substitute a browser string anywhere.
UnlobBot/1.0 (+https://www.unlob.com/crawler)Match on the token UnlobBot in robots.txt — it is stable across versions, so a rule you write today keeps working after we bump the number.
Verify before you trust it
Anyone can copy a user-agent string, so treat ours as a claim rather than as proof, and never grant access on the strength of it. Confirm with a reverse DNS lookup on the requesting address followed by a forward lookup on the result — the same double-lookup Google and Bing document.
$ dig -x 203.0.113.42 +short
crawl-042.unlob.com.
$ dig +short crawl-042.unlob.com
203.0.113.42 # forward lookup matches — genuineWhat we do with your response
Stated as a contract, so you can predict our behaviour rather than infer it from your logs. Rate-limiting us is a legitimate and effective signal — you do not have to block us outright to slow us down.
| Status | Meaning | What UnlobBot does |
|---|---|---|
| 200 | OK | Text is extracted and the ETag and Last-Modified are stored for the next conditional request. |
| 304 | Not Modified | Nothing is transferred and nothing is reprocessed. This is the common case on a recrawl, and it is why we cost you almost no bandwidth. |
| 301 / 308 | Permanent redirect | Followed, up to five hops, and the target replaces the source in our frontier. |
| 302 / 307 | Temporary redirect | Followed, but the original URL is kept as canonical. |
| 404 / 410 | Gone | The URL is dropped. A 410 is treated as final and removes it from the index with a typed reason, readable through why_not. |
| 429 | Too Many Requests | Exponential backoff, honouring Retry-After. Sustained 429s lower our rate for that host beyond the immediate retry. |
| 5xx | Server error | Backed off and retried later. Repeated failures reduce crawl frequency rather than increasing it. |
We also honour noindex and nofollow, both as a robots meta tag and as an X-Robots-Tag header, and we read Sitemap: directives from robots.txt. A maintained sitemap with accurate lastmod values measurably reduces how much we fetch from you, because it lets us skip everything unchanged.
Controlling UnlobBot
Block entirely
User-agent: UnlobBot
Disallow: /Restrict paths and slow us down
User-agent: UnlobBot
Disallow: /admin/
Disallow: /internal/
Crawl-delay: 5
# Everything else is allowed.Rules take effect on the next fetch of your robots.txt. If you need something removed immediately, email hello@unlob.com — we will tombstone the source, which records it in the removal ledger sowhy_not reports it as removed rather than never seen.
What we do and do not do
We do
- Honour robots.txt on every fetch, with longest-match precedence.
- Respect
Crawl-delaywhere it is set. - Identify ourselves with a stable user agent that resolves to this page.
- Hold to roughly one request per second per host.
- Use conditional requests, so unchanged pages cost you almost nothing.
- Honour
noindexandX-Robots-Tag. - Back off on
429and5xx, honouringRetry-After. - Read your sitemap, and use
lastmodto fetch less. - Accept coverage gaps where we are blocked, and report them honestly.
We do not
- Train AI models on what we crawl, or sell it as a training corpus.
- Bypass paywalls, logins or anti-bot measures.
- Impersonate a browser or rotate identities to avoid rate limits.
- Distribute a crawl across addresses to exceed a per-host limit.
- Run a second crawler under another name for another purpose.
- Crawl regions of the web nobody queries, purely to have them.
We do not train models on your content
This is the question publishers actually want answered in 2026, so here it is without hedging. We do not train models on crawled content, and we do not sell it as a training corpus. UnlobBot exists to build a search index, and a search index is the thing that sends people back to you.
Results link to you
A search result is a url, title, snippet and score. The url is the point — an agent that wants to read the page has to go and get it.
No bulk export
Full text is served one document at a time, as a metered and authenticated API call. There is no endpoint that hands anyone your site in bulk.
One crawler, one purpose
UnlobBot is our only crawler. Blocking it blocks everything we do — there is no second agent under a different name collecting for a different purpose.
Why we crawl less than you might expect
Most crawlers try to fetch as much of the web as they can afford. We do not: our index is deliberately bounded, so we fetch where there is demonstrated demand rather than speculatively. In practice that means fewer requests to your server, and the ones we do make are for content someone actually went looking for.
Freshness works the same way. Conditional requests mean an unchanged page returns 304 and transfers no content, so keeping our index current does not mean periodically re-downloading your site. If your logs show us fetching something repeatedly and getting a 200 each time, anETag or Last-Modified header on that route will stop it — and if it does not, email us, because that is a bug on our side.
Frequently asked questions
What is UnlobBot?
The web crawler that builds the unlob search index. It discovers and fetches pages, extracts the readable text, and stores passages from it so they can be retrieved by search.
What user-agent string does UnlobBot send?
UnlobBot/1.0 (+https://www.unlob.com/crawler) — on every request, including the fetch of your robots.txt. We do not rotate it, shorten it or substitute a browser string. Match on the token UnlobBot in your robots.txt; it is stable across versions.
How do I block UnlobBot?
Add a Disallow rule for UnlobBot in your robots.txt. We honour it, including crawl-delay, with longest-match precedence and support for wildcards and end-anchors. Blocking takes effect on our next fetch of your robots.txt, which is at most 24 hours later.
How fast does UnlobBot crawl?
Roughly one request per second per host, and slower where robots.txt sets a crawl-delay. We do not run distributed crawls against a single host to circumvent the limit, and we do not parallelise across subdomains to get around a per-host budget.
How do I verify a request really came from UnlobBot?
By reverse DNS on the requesting IP, then a forward lookup on the result — the same double-lookup Google and Bing document. Anyone can copy our user-agent string, so treat the user-agent alone as a claim rather than as proof, and never grant access on the strength of it.
What happens if you cannot fetch our robots.txt?
A 404 or 410 is treated as no restrictions. A 5xx, a timeout or a connection failure is treated as a full disallow until we can read it again — we do not crawl a site whose rules we could not confirm. Either way the politeness limits still apply, and a server failing to serve robots.txt is usually a server under stress.
How do you respond to 429 and 503?
We back off exponentially and honour Retry-After when it is present. Sustained 429 or 503 from a host reduces our crawl rate for that host well beyond the immediate retry. Rate-limiting us is a legitimate and effective signal; you do not need to block us outright to slow us down.
Do you respect noindex and nofollow?
Yes — both the robots meta tag and the X-Robots-Tag response header. A page carrying noindex is fetched, evaluated and then not indexed. If you would rather we did not fetch it at all, use robots.txt, which is cheaper for both of us.
Do you use crawled content to train AI models?
No. We do not train models on crawled content and we do not sell it as a training corpus. The index stores extracted passages so they can be retrieved and cited back to you with a link — search results, not a substitute for your page. An agent that wants the full text of a result fetches it through us, which is a metered API call rather than a bulk export.
Do you bypass paywalls or anti-bot measures?
No. Where a site blocks us we accept the coverage gap and surface it through why_not rather than escalating. It is the cheaper policy and it is the correct one — an arms race costs both parties money and produces worse data.
How do I request removal of content?
Email us and we will tombstone the source, which records it in the removal ledger so why_not reports it as removed rather than never seen. Adding a Disallow rule to robots.txt also removes it on the next crawl cycle. We do not require you to prove ownership for a removal request about your own domain.
Does UnlobBot execute JavaScript?
Only where a page genuinely does not render without it. Most fetches are a single plain HTTP request, which is the cheapest path for your server as well as ours, so we prefer it.
Do you support sitemaps?
Yes. We read Sitemap directives from robots.txt and use lastmod to decide what is worth refetching. A well-maintained sitemap measurably reduces how much we crawl your site, because it lets us skip everything that has not changed.
Questions about UnlobBot?
Email us. Removal requests, crawl-rate concerns and coverage questions all get a real answer from a person.