What is Web crawler?
Verified 5 August 2026
The component that acquires the corpus. It maintains a frontier of URLs to fetch, applies politeness constraints per host, and feeds an extraction pipeline.
Crawling politely is a constraint on throughput and an ethical obligation: honour robots.txt including crawl-delay, identify yourself with a real user agent, and hold to roughly one request per second per host. Where a site blocks you, the correct response is to accept the gap rather than escalate.
How unlob handles this: UnlobBot
Common questions
What is Web crawler?
A web crawler discovers and fetches pages by following links, respecting robots.txt and rate limits.
How does Web crawler work in practice?
Crawling politely is a constraint on throughput and an ethical obligation: honour robots.txt including crawl-delay, identify yourself with a real user agent, and hold to roughly one request per second per host. Where a site blocks you, the correct response is to accept the gap rather than escalate.
See it working
The free tier is 10,000 requests a month with no card — enough to test any of this against your own queries.