Monitoring topics and entities
Teams tracking coverage of a topic, company or event over time.
The problem
- Monitoring by repeated search returns the same items every run, because relevance does not know what you saw last time.
- Syndication means one event fills the feed. Forty copies of a press release is not forty things happening.
- Crawl time and publication time get confused, so feeds either miss recently published older material or re-surface things already seen.
The approach
- browse needs no query at all — it returns the most recent passages, optionally within a vertical, which is what a monitoring loop actually wants.
- published_from set to the previous run timestamp is the whole trick to a no-duplicates feed.
- collapse=story converts syndication into distinct events, with a group size showing how widely each was carried.
The parameters that matter
| Parameter | Value | Why |
|---|---|---|
published_from | last run timestamp | Only material published since the previous run. |
collapse | story | Distinct events rather than syndicated copies. |
sort | published | Newest first, by content date rather than crawl date. |
min_independent_sources | 2 | Filters single-source noise out of the feed. |
# Everything published since the last run
curl -H "x-api-key: $UNLOB_API_KEY" \
"https://api.unlob.com/search?q=semiconductor+export+controls&published_from=1754352000&collapse=story&sort=published"
# Or with no query at all
curl -H "x-api-key: $UNLOB_API_KEY" "https://api.unlob.com/browse?sort=recency&limit=20"Where this is not the right tool
- Not every page carries a reliable publication date. Passages without one are excluded when published_from is set, which narrows the pool.
- Freshness depends on crawl cadence. For breaking news within minutes, a dedicated news feed will beat any general web index.
Frequently asked questions
How fresh is the index?
The delta crawler prioritises change detection and demand, so actively-queried areas refresh faster than the long tail. Both fetched_at and published_at are exposed so you can see exactly how current any result is.
How do I avoid duplicates between runs?
Set published_from to the previous run timestamp and collapse=story. The first prevents re-surfacing, the second prevents one event filling the feed.
Build it on the free tier
10,000 requests a month, no card. Every parameter above works on every plan.