Work through more results than one call returns

There is no cursor. Partition the query space by date window, host or content type and run one call per partition — which is faster and cheaper than paging anyway.

The problem

The reflex is to look for `offset` or a cursor. This API has neither, deliberately: deep pagination over a relevance ranking returns steadily worse results while costing the same per call, and an agent working through page nine of anything has usually asked the wrong question.

What to send

limit

up to your plan ceiling

Raise it before you reach for anything cleverer.

published_from / published_to

a moving window

Partition by time. The most reliable way to walk a large space exhaustively.

site / content_type[]

one per call

Partition by source or kind, when time is not the right axis.

The code

Partition by monthpython

import httpx
from datetime import datetime, timezone, timedelta

HEADERS = {"x-api-key": "ulb_..."}
API = "https://api.unlob.com/search"

def months(start: datetime, end: datetime):
    cur = start
    while cur < end:
        nxt = (cur + timedelta(days=32)).replace(day=1)
        yield cur, min(nxt, end)
        cur = nxt

seen, out = set(), []
start = datetime(2025, 1, 1, tzinfo=timezone.utc)
end   = datetime(2026, 1, 1, tzinfo=timezone.utc)

for lo, hi in months(start, end):
    r = httpx.get(API, headers=HEADERS, params={
        "q": "semiconductor export controls",
        "published_from": int(lo.timestamp()),
        "published_to": int(hi.timestamp()),
        "limit": 25,
    }).json()
    # Windows can overlap at the boundary; dedupe on id, not url.
    for hit in r["results"]:
        if hit["id"] not in seen:
            seen.add(hit["id"])
            out.append(hit)

print(len(out), "unique results across", 12, "partitions")

The mistake to avoid

Deduplicating partitions by URL rather than by `id`. The same passage can surface under two URLs, and boundary overlap between windows is normal — `id` is the stable identity and the only one worth keying on.

Frequently asked questions

Why is there no offset parameter?

Because deep paging over a relevance ranking degrades quickly and costs the same per call. Partitioning gives you exhaustive coverage of a defined space, runs in parallel, and each call returns its own best results rather than someone else's leftovers.

How do I know I have covered everything?

You cannot, from result counts alone — that is what `why_not` is for. If a specific document has to be in your set, ask about it directly rather than inferring its absence from a count.

Try it against your own queries

10,000 requests a month on the free tier, no card. Enough to run a real evaluation set rather than a demo.

Get an API key All recipes