`

Unlob Bot

Information about our web crawler

About UnlobBot

UnlobBot is the web crawler used by Unlob search engine to analyze web content and provide intelligent search results. Our bot respects robots.txt files and follows ethical crawling practices.

How to Identify Our Bot

User-Agent: UnlobBot/1.0 (+https://unlob.com/bot)

Our crawler can be identified by this user agent string in your server logs.

Crawling Policies

  • ✓ We respect robots.txt directives
  • ✓ We follow a reasonable crawl rate
  • ✓ We only index publicly accessible content
  • ✓ We honor noindex and nofollow tags

Control Unlob Bot

Using robots.txt

User-agent: UnlobBot
Disallow: /private/
Disallow: /admin/

Using Meta Tags

<meta name="robots" content="noindex">

Contact Us

If you have any questions or concerns about our crawler, please contact us at: [email protected]