Monitor a topic without seeing the same story twice

Combine collapse=story with published_from to poll a topic and receive one row per story rather than one row per outlet that syndicated it.

The problem

A naive poll returns the same event twenty times, because twenty outlets ran the same wire copy. Deduplicating by URL does not help — the URLs are all different. Deduplicating by title barely helps, because every outlet rewrites the headline.

What to send

collapse

story

One row per story cluster, not per document. The syndication problem, solved server-side.

published_from

unix timestamp

The publication clock, so you get what was written since you last looked.

sort

recency

Newest first, which is what a monitor wants.

The code

A polling looppython

import time, httpx

API = "https://api.unlob.com/search"
HEADERS = {"x-api-key": "ulb_..."}
seen: set[str] = set()
since = int(time.time()) - 86_400   # first run: last 24h

while True:
    r = httpx.get(API, headers=HEADERS, params={
        "q": "central bank interest rate decision",
        "collapse": "story",     # one row per story, not per outlet
        "published_from": since,
        "sort": "recency",
        "limit": 25,
    })
    hits = r.json()["results"]

    for hit in hits:
        # collapse handles syndication; this guards against a story that
        # keeps developing and legitimately reappears in a later window.
        if hit["id"] in seen:
            continue
        seen.add(hit["id"])
        print(hit["published_at"], hit["title"], hit["url"])

    # Advance the clock to the newest thing we actually saw, never to
    # now() — anything published during the request would be skipped.
    if hits:
        since = max(h["published_at"] for h in hits) + 1
    time.sleep(300)

The mistake to avoid

Advancing your cursor to `now()` after each poll. Anything published between the server building your response and your clock reading is skipped, permanently and silently. Advance to the newest `published_at` you actually received.

Frequently asked questions

What is the difference between collapse=story and collapse=host?

`story` groups documents that report the same event across different publishers. `host` keeps one result per domain regardless of subject. For monitoring you almost always want `story`; for source diversity in a research context, `host`.

How often should I poll?

Match it to how fast the topic actually moves, and remember the rate limit is per plan. Polling a slow topic every thirty seconds spends quota to receive the same empty result; five minutes is ample for most news and an hour is fine for most everything else.

Try it against your own queries

10,000 requests a month on the free tier, no card. Enough to run a real evaluation set rather than a demo.

Get an API key All recipes