listen · your own sources
Pull connectors
Feeds and APIs VibeDraft polls.
In Your own sources
A pull connector is a source VibeDraft goes and reads on a schedule: a feed, a search or an API. Every run asks only for what's new since the last one, writes each item once, and leaves a run row behind so a broken feed is visible the same day.
schedule · when to read → pull connector · fetch what's new → item · stored raw → detector · does it match?
rule 1 — Ask for what's new, not everything · Every run carries a cursor: an ETag, a last-seen id or a timestamp. Re-reading the whole feed wastes quota and makes duplicates.
rule 2 — Write each item once · Upsert on a dedupe key (the feed's guid, the video id, the post id). A second sighting updates the row; it never makes a new signal.
rule 3 — A run always leaves a row · Success, partial or error, with counts and cost. Silence is the one result that's never allowed.
what it is
A scheduled fetch with a memory.
In data-engineering terms this is incremental extraction: a scheduler triggers a sync, the connector reads from the source's API or feed starting at a watermark, and writes new records into a raw landing table. It's the opposite of push, where the source sends items to you. Most public sources only offer pull.
Poll interval — cadence · schedule · How often a source is read. Set by how fast its items go stale, not by how often it posts.
Cursor — watermark · since_id · Where the last run stopped: the newest id, timestamp or page token seen.
Conditional GET — ETag · If-Modified-Since · Ask the server whether anything changed. A 304 Not Modified costs almost nothing.
Backfill — initial sync · history load · The first run reads history. Those items are stored but marked so old posts never become fresh signals.
Backoff — rate-limit handling · On a 429 or a 5xx, wait and retry later instead of hammering. The wait is recorded as resume_at.
Full-text fetch — enrichment · Many feeds carry only a summary. A second request fetches the article body so detectors read the whole thing.
how it works
Seven steps per run, the same for every kind.
The loop below is the same whether the source is an RSS feed, a Hacker News search or the X API. Only the fetch step changes per kind.
- Scheduler — cron tick
- Claim due sources — next_run_at <= now
- Fetch — with cursor / ETag
- Parse — to item shape
- Dedupe + upsert — on the dedupe key
- Advance cursor — only after write
- Run row — counts · status · cost
- Notify readers — item.created events
Claim the sources that are due
Fetch with the cursor
Send the stored ETag or Last-Modified for feeds, a since_id or start_time for APIs, a page token for paginated searches. A 304 ends the run as a success with zero new items.
Parse to the item shape
Map whatever came back (XML, JSON, HTML) into one item record: external id, url, author, published time, title, text, raw payload.
Dedupe and upsert
Write on the dedupe key. Same key and same content hash: skip. Same key, new hash: update the row and note the edit.
Advance the cursor only after the write
If the run dies mid-way, the next run starts from the old cursor and the dedupe key absorbs the overlap. Nothing is lost and nothing doubles.
Write the run row
Started, finished, status (success, partial, error), items seen, new, updated, the new cursor, the error text and the API cost.
Tell the readers
Emit an item event so detectors, drafting and research pick up new items without polling the store themselves.
the object
A connector config and a run.
The source row from the parent page carries a config; for pull kinds this is its shape. Each run writes the second object.
One row per fetch. The health page reads these.
| Field | Type | What it holds |
|---|---|---|
run_id | uuid | Joins items to the fetch that brought them |
source_id | uuid | Which connector |
started_at · finished_at | timestamp | Duration comes from these |
status | enum | success, partial (some feeds or pages failed), error |
items_seen | int | Everything the fetch returned |
items_new · items_updated | int | After dedupe |
cursor_before · cursor_after | text | So any run can be replayed |
http_status | int | null | 304s are normal and cheap |
error | text | null | The message, verbatim |
cost_usd | numeric | Paid reads (X search, enrichment) per run |
one run, as data
pull config
{
"kind": "rss",
"target": "https://example-competitor.com/blog/feed.xml",
"auth_ref": null,
"cadence_min": 60,
"cursor": "2026-10-16T22:06:37Z",
"etag": "\"6f1a-5c2\"",
"last_modified": "Fri, 16 Oct 2026 22:06:37 GMT",
"full_text": true,
"backfill_days": 30,
"max_staleness_h": 24,
"next_run_at": "2026-10-17T05:00:00Z"
}run
{
"run_id": "run_8c1e",
"source_id": "src_rss_competitor_blog",
"started_at": "2026-10-17T04:00:02Z",
"finished_at": "2026-10-17T04:00:05Z",
"status": "success",
"items_seen": 20,
"items_new": 1,
"items_updated": 0,
"cursor_before": "2026-10-15T14:00:00Z",
"cursor_after": "2026-10-16T22:06:37Z",
"http_status": 200,
"error": null,
"cost_usd": 0
}the pull catalog
Sixteen kinds of pull, and what each needs.
Each card is one connector kind: how it fetches, its dedupe key and cursor, a sensible cadence, and the catch. Cadences are starting points; the health page explains how to tune them. Check each platform's current terms and pricing before automating it.
Feeds and pages — the open web
pull.rss — cloud
RSS or Atom feed
Read the feed with a conditional GET; fetch full text when it only carries a summary.
- dedupe
- the item guid, else the link
- cursor
- ETag + Last-Modified
- cadence
- hourly
- catch
- guids change when a site migrates; a content hash catches the repeats
pull.page_diff — cloud
A page with no feed
Fetch the page, strip the chrome, and store a new item when the text changes: pricing pages, changelogs, docs.
- dedupe
- url + content hash
- cursor
- last hash
- cadence
- daily
- catch
- cookie banners and dates change every load; diff the main content only
pull.newsletter_archive — cloud
A newsletter's web archive
Most newsletter platforms publish an RSS feed of past issues. Pulling it avoids a forwarding address entirely.
- dedupe
- issue url
- cursor
- ETag
- cadence
- daily
- catch
- paid issues show only a teaser; use the forwarding address for those
pull.youtube_channel — cloud
YouTube channel
Poll the channel for new videos, then take the captions as the transcript; transcribe the audio when there are none.
- dedupe
- video id
- cursor
- newest published time
- cadence
- every 6 hours
- catch
- auto-captions are rough on names; transcription costs per minute
pull.podcast — cloud
Podcast
Read the show's RSS feed, download the audio enclosure, transcribe it, store the transcript as the item text.
- dedupe
- enclosure guid
- cursor
- ETag
- cadence
- daily
- catch
- an hour of audio is a long item; chunk it before detectors read it
Social platforms — APIs with sign-in or paid reads
pull.x_search — cloud
X search and mentions
Recent-search queries for pain words, plus replies and mentions of your own handle.
- dedupe
- post id
- cursor
- since_id
- cadence
- every 15 to 60 min
- catch
- reads are paid; log cost per run and cap it
pull.x_bookmarks — cloud
X bookmarks
Read the founder's bookmarks folder by folder with their sign-in.
- dedupe
- post id + folder
- cursor
- pagination token
- cadence
- daily
- catch
- needs the user's OAuth; bookmarks are private, keep them per account
Bookmark folders can be listed too; they are usually not worth storing as their own source.
pull.threads_search — cloud
Threads keyword search
Search public Threads posts for pain words.
- dedupe
- media id
- cursor
- timestamp
- cadence
- hourly
- catch
- search access is limited by the API's permissions
pull.ig_inbound — cloud
Instagram comments and DMs
Read comments on your posts and messages to your account.
- dedupe
- comment or message id
- cursor
- timestamp
- cadence
- every 20 min
- catch
- needs the comments and messages permissions; some accounts need a reconnect first
Communities and launches — where buyers ask and builders ship
pull.hn_search — cloud
Hacker News
Keyword search and Show HN through the Algolia API.
- dedupe
- object id
- cursor
- created_at_i
- cadence
- hourly
- catch
- comments and stories are separate types; read both
pull.reddit_feed — cloud
Reddit subreddits and searches
Subreddit and search feeds for asks and complaints. Reddit ends RSS feeds on Nov 13, 2026 and closes public API access in March 2027 (PPC Land).
- dedupe
- post id (t3_…)
- cursor
- newest created time
- cadence
- hourly
pull.product_hunt — cloud
Product Hunt launches
Daily launches in your category, and comments on competitors' launches.
- dedupe
- post id
- cursor
- day
- cadence
- every 6 hours on launch days
- catch
- comments are where buyers compare; pull them, not just the listing
pull.trustmrr — cloud
TrustMRR listings
Startups with verified revenue: new listings and revenue changes.
- dedupe
- listing id
- cursor
- last seen listing
- cadence
- daily
- catch
- read gently and check the site's terms first
Products and reviews — code, changelogs and review sites
pull.github — cloud
GitHub repos
Releases and issues on repos you follow: competitors, integrations, your own.
- dedupe
- release or issue id
- cursor
- ETag + since
- cadence
- every 6 hours
- catch
- issues are noisy; filter by label or reactions
pull.reviews — cloud
Review sites and app stores
New reviews on your listing and your competitors': G2, Capterra, the app stores.
- dedupe
- review id
- cursor
- newest review date
- cadence
- daily
- catch
- use official feeds or vendor APIs where you have access; many sites forbid scraping
pull.changelog — cloud
Competitor changelogs
A page-diff or feed on each competitor's changelog and pricing page.
- dedupe
- url + content hash
- cursor
- last hash
- cadence
- daily
- catch
- price changes are triggers; send them to a detector, not just research
failure modes
How pulls break, and how to notice.
The feed moved — The url now returns 301 or 404. Runs succeed with zero items and nobody notices. · catch itTreat a run with an error status or zero items past the max staleness as stalled, and alert.
Summary-only feeds — Detectors read two lines and miss the point of the post. · catch itfull_text on, with the extracted body stored next to the raw payload.
Guids that change — A site migration re-issues every guid and floods detectors with old posts. · catch itA content-hash check on upsert, and a published-time filter on signals.
Rate limits — 429s mid-run leave half a page read. · catch itBack off with resume_at; advance the cursor only after the write.
Backfill floods — The first run of a new source makes 500 "fresh" signals from last month. · catch itMark backfilled items; only items inside the trigger's freshness window can become signals.
Paid reads creep — An X search query broadens and the bill grows quietly. · catch itcost_usd on every run, and a per-source monthly cap.
A route shuts down — Reddit's public RSS ends Nov 13, 2026. · catch itKeep the item shape; record the switch date so metrics can be split before and after.
data to capture
What every pull records.
The same four lines every stage uses, under the guide's eight data rules: log the decision, version what decided, freeze the inputs, keep the negatives, log the odds, every human touch is a label, one chain of ids, keep the raw.
record
- one run row per fetch, with cursor before and after, counts, status, http status and cost
- every item raw, with published and captured time and a content hash
- edits (same key, new hash) and deletions at the source, with the time
- which detectors read each new item
label
- a source whose items became signals the founder marked real: useful
- runs the founder flagged as broken: true outages
- a source the founder disconnected: not worth it
trains
- cadence per source from its own posting rhythm
- yield per source per dollar: signals, leads, replies
- an anomaly check on items_new per run
test
- cadence (hourly vs every 6 hours) · A/B by source, measured on signal freshness and cost
- a new source for a trigger · shadow beside the existing ones, compare matches
- full text on or off · offline replay of detector precision