What is robots.txt?
Verified 5 August 2026
The convention since 1994. Allow and Disallow rules are matched by longest prefix, with wildcards and end-anchors; Crawl-delay sets a minimum interval between requests.
It is voluntary, which makes honouring it a statement about how a crawler operates. A missing or unreachable robots.txt is conventionally treated as permission, and a crawler should fail open rather than hammering a site that cannot serve the file.
How unlob handles this: UnlobBot
Common questions
What is robots.txt?
robots.txt is a file at a site’s root telling crawlers which paths they may fetch and how fast.
How does robots.txt work in practice?
It is voluntary, which makes honouring it a statement about how a crawler operates. A missing or unreachable robots.txt is conventionally treated as permission, and a crawler should fail open rather than hammering a site that cannot serve the file.
See it working
The free tier is 10,000 requests a month with no card — enough to test any of this against your own queries.