Use case
Monitoring topics and entities
Teams tracking coverage of a topic, company or event over time.
The problem, and the approach
The problem
- Monitoring by repeated search returns the same items every run, because relevance does not know what you saw last time.
- Syndication means one event fills the feed. Forty copies of a press release is not forty things happening.
- Crawl time and publication time get confused, so feeds either miss recently published older material or re-surface things already seen.
The approach
- browse needs no query at all — it returns the most recent passages, optionally within a vertical, which is what a monitoring loop actually wants.
- published_from set to the previous run timestamp is the whole trick to a no-duplicates feed.
- collapse=story converts syndication into distinct events, with a group size showing how widely each was carried.
The parameters that matter
| Parameter | Value | Why |
|---|---|---|
published_from | last run timestamp | Only material published since the previous run. |
collapse | story | Distinct events rather than syndicated copies. |
sort | published | Newest first, by content date rather than crawl date. |
min_independent_sources | 2 | Filters single-source noise out of the feed. |
# Everything published since the last run
curl -H "x-api-key: $UNLOB_API_KEY" \
"https://api.unlob.com/search?q=semiconductor+export+controls&published_from=1754352000&collapse=story&sort=published"
# Or with no query at all
curl -H "x-api-key: $UNLOB_API_KEY" "https://api.unlob.com/browse?sort=recency&limit=20"A monitoring feed is a different query shape
Monitoring is not repeated searching. A search asks which passages best match a question; monitoring asks what is new in a slice of the index since the last time it was asked, and the second question does not need a query string at all. Browsing a slice by recency, bounded by publication date, gives a feed that is complete over that slice rather than ranked within it — and completeness is the property a monitoring workflow is actually buying.
Ranking is in fact harmful here. A relevance-ordered feed silently omits the item that matched weakly, and the whole point of monitoring is to notice the thing nobody thought to ask about. Take the slice, collapse it, and let the downstream classifier decide what matters.
The date fields are where monitoring goes wrong
Republished and syndicated copies frequently carry a later timestamp than the original reporting, so sorting a news slice by date fills the top with the syndication tail. Collapsing to one representative per story cluster fixes it, and the corroboration count survives the collapse, so an alert can still say how widely something was carried.
The subtler problem is timestamp churn: a page whose date field updates on every edit re-enters the window repeatedly without its substance changing, which inflates any count built over that window. If the output is an alert rather than an answer, deduplicate on content and treat the date as a hint. And accept the floor — a general web index sees a story when it crawls it, so anything that needs minutes rather than hours belongs on a dedicated wire feed, with this as the wider net behind it.
Where this is not the right tool
- Not every page carries a reliable publication date. Passages without one are excluded when published_from is set, which narrows the pool.
- Freshness depends on crawl cadence. For breaking news within minutes, a dedicated news feed will beat any general web index.
Frequently asked questions
How fresh is the index?
The delta crawler prioritises change detection and demand, so actively-queried areas refresh faster than the long tail. Both fetched_at and published_at are exposed so you can see exactly how current any result is.
How do I avoid duplicates between runs?
Set published_from to the previous run timestamp and collapse=story. The first prevents re-surfacing, the second prevents one event filling the feed.
Build it on the free tier
10,000 credits a month, no card. Every parameter above works on every plan.