phase 2 · listen
Your own sources
Go deeper
The feeds, scrapers and saves a founder already lives in: the newsletters they read, the channels they watch, the posts they bookmark. Each one becomes a connector with a health check, every item lands raw and append-only, and detectors, drafting and research all read from that one store.
source · where items come from → detector · the rule that watches → signal · one person caught → lead · fits and is ready
rule 1 — Raw first, then derived · Store each item exactly as it arrived, with its url and both timestamps. Search indexes, entities and matches are built on top and never edit it.
rule 2 — Every source has a health check · A max staleness and a run log for every connector, so a quiet feed reads as "stalled", never as "nothing happened".
rule 3 — Official routes, your own saves · APIs and data exports first. Anything that needs the user's session runs on their machine and reads only what they saved.
what it is
A source is a configured place items come from.
In data-engineering terms a source is a connector (or ingestion source): a configured pull or push that writes records into a raw landing table. Each fetch is a run; each record is an item; anything computed from items (chunks, embeddings, entities, matches) is derived. The Detector catalog decides what counts as a match; this page decides what gets read in the first place.
runs on
cloud
Public feeds and APIs read on a schedule from the server. RSS, YouTube, Hacker News, Reddit, GitHub, X bookmarks with the user's sign-in.
runs on
your machine — needs your session
Anything only visible while signed in, read in the user's own browser or desktop app.
runs on
import — a file you upload
A platform's data download: Instagram, TikTok and X all export saved and liked items as files.
runs on
push
A forwarding address for newsletters, a share sheet, an email-to-save address or a save command in chat.
Connector — ingestion source · One configured source: kind, config, sign-in, schedule.
Run — sync · job run · One fetch of one source, with counts, a cursor and a status.
Item — raw record · landing row · One post, video, email or save, stored as it arrived.
Cursor — watermark · ETag · Where the last run stopped, so the next one only reads what's new.
Dedupe key — natural key · The id that makes the same item land once: a guid, a video id, a Message-ID.
Max staleness — freshness SLO · How long a source can go without a new item before it's flagged.
the objects
Four records.
A source owns many runs; a run writes many items; an item gets many derived rows. Everything carries the account id, so one founder's sources never mix with another's.
source
What to watch. One row per connector the founder sets up.
| Field | Type | What it holds |
|---|---|---|
id | string | Stable: src.<kind>.<slug>. Every run and item carries it. |
account_id | uuid | The founder's account. Sources are never shared across accounts. |
kind | connector kind | One of the library's kinds: rss, newsletter, youtube_playlist, x_bookmarks… |
config | object | What to read: a url, a query, a folder, a playlist or a handle. |
auth_ref | reference | null | Points to a stored token or a device session. Never the secret itself. |
runs_on | cloud | device | import | push | Who does the fetching. |
schedule | cron | on arrival | How often it runs; push and import sources run on arrival. |
max_staleness_h | hours | Longer than this with no new item and the source is stalled. |
retention_days | days | null | How long items are kept. Null keeps them until disconnect. |
feeds | reader list | Which readers take its items: detectors, drafting, research, labels. |
status | active | paused | disconnected | Paused keeps items; disconnected deletes them. |
version | integer | Bumped when the config or schedule changes; stamped on every run. |
one source, as data
{
"id": "src.newsletter.founder-inbox",
"account_id": "acct_7f3c…",
"kind": "newsletter",
"config": {
"forward_to": "in+7f3c@inbox.example.com",
"senders": [
"*"
]
},
"auth_ref": null,
"runs_on": "push",
"schedule": "on arrival",
"max_staleness_h": 168,
"retention_days": null,
"feeds": [
"detectors",
"drafting",
"research"
],
"status": "active",
"version": 2
}run
Each fetch. The run log is how a quiet source and a broken one get told apart.
| Field | Type | What it holds |
|---|---|---|
run_id | string | Unique per fetch. |
source_id | string | The source it ran for, with the source version it used. |
runner | string | cloud, or the device name for session-bound sources. |
started_at · finished_at | timestamps | Duration comes free. |
status | success | partial | error | Partial means some feeds or pages failed and some didn't. |
items_seen · items_new · items_updated | integers | Seen counts everything read; new and updated are what got written. |
cursor | string | null | ETag, Last-Modified, a since-id or a page token: where the next run starts. |
error | text | null | The first failing reason, for partial and error runs. |
cost_usd | number | API, transcription or proxy cost for this run. |
one run, as data
{
"run_id": "run_2026-10-17T04:07:22Z_rss",
"source_id": "src.rss.indie-blogs",
"source_version": 3,
"runner": "cloud",
"started_at": "2026-10-17T04:07:22Z",
"finished_at": "2026-10-17T04:08:01Z",
"status": "partial",
"items_seen": 211,
"items_new": 4,
"items_updated": 1,
"cursor": "W/\"5f2a-1c9\"",
"error": "3 of 22 feeds returned 404",
"cost_usd": 0
}item
What came in, stored as it arrived, append-only. If the source deletes it, it's marked gone, not removed.
| Field | Type | What it holds |
|---|---|---|
item_id | string | Unique per stored item. |
source_id · run_id | strings | Where and when it was collected. |
external_id | string | The dedupe key: a guid, video id, tweet id or Message-ID. Unique per source. |
url | url | The canonical link back to the original. |
author | object | Handle, name and profile url, as shown at capture. |
published_at · captured_at | timestamps | When it was posted, and when it was caught. The gap is detection lag. |
title · body | text | Full text, the transcript for video and audio, or the email body. |
media | list | Links to images, video or audio, not copies. |
raw | json | The payload exactly as received, for re-parsing later. |
content_hash | string | Detects edits; an edited item gets a new version row. |
gone_at | timestamp | null | Set when the original disappears at the source. |
one item, as data
{
"item_id": "itm_01JAX…",
"source_id": "src.youtube.ai-playlist",
"run_id": "run_2026-10-16T06:30Z_yt",
"external_id": "yt:dQ4w…",
"url": "https://www.youtube.com/watch?v=dQ4w…",
"author": {
"handle": "@somechannel",
"name": "Some Channel"
},
"published_at": "2026-10-16T00:00:00Z",
"captured_at": "2026-10-16T06:30:45Z",
"title": "How we got our first 100 users",
"body": "[transcript, 9,400 words]",
"media": [],
"raw": {
"…": "as received"
},
"content_hash": "sha256:9c1e…",
"gone_at": null
}derived
Everything computed from items. Rebuildable at any time, so the item stays the source of truth.
| Field | Type | What it holds |
|---|---|---|
derived_id | string | Unique per row. |
item_id | string | The item it was built from. Many rows per item. |
type | chunk | embedding | entity | topic | match | label | What was computed. |
value | json | The chunk text, vector ref, entity name, topic, detector match or label. |
by | string | The model, prompt or detector that produced it, with its version. |
created_at | timestamp | When it was computed. Re-running a newer version adds rows; it never edits old ones. |
one derived row, as data
{
"derived_id": "drv_01JAY…",
"item_id": "itm_01JAX…",
"type": "match",
"value": {
"detector_id": "det.tried-it-and-it-failed.youtube",
"detector_version": 2,
"verdict": "real",
"signal_id": "sig_01JAZ…"
},
"by": "judge@prompt-v7",
"created_at": "2026-10-16T06:31:10Z"
}source → run → item → derived → detectors · drafting · research · labels
the connector library
Twenty sources a founder can bring.
Each card is a connector kind: how items get in, where it runs, what sign-in it needs, how often it runs, the max staleness that flags it, the key that dedupes it, and what its items feed.
What you read — newsletters, blogs, video, audio
cloud
RSS and Atom feeds
Poll each feed with ETag and Last-Modified; fetch the full article when the feed only carries a summary.
- sign-in
- none
- cadence
- hourly
- stale at
- 24 h
- dedupe
the entry's guid, else its link- feeds
detectors — research · drafting
cloud
Blogs without a feed
Fetch the post index, diff it against the last run, pull new posts.
- sign-in
- none
- cadence
- daily
- stale at
- 72 h
- dedupe
canonical url- feeds
research — detectors
push
cloud
Newsletters
A private forwarding address per account, or a Gmail label read with the user's sign-in (read-only).
- sign-in
- none, or Gmail read-only
- cadence
- on arrival
- stale at
- 7 days
- dedupe
Message-ID- feeds
detectors — drafting · research
cloud
your machine
YouTube channels and playlists
Poll the channel or playlist; captions become the transcript, speech-to-text when there are none.
- sign-in
- API key
- cadence
- every 6 h
- stale at
- 96 h
- dedupe
video id- feeds
research — drafting
cloud
Podcasts
Read the show's RSS; transcribe each new episode's audio.
- sign-in
- none
- cadence
- daily
- stale at
- 7 days
- dedupe
enclosure guid- feeds
research — drafting
cloud
Papers
- sign-in
- none
- cadence
- daily
- stale at
- 72 h
- dedupe
paper id- feeds
research
Communities and launches — where buyers ask and builders launch
cloud
Hacker News
Algolia's HN Search API: keywords, Ask HN, Show HN.
- sign-in
- none
- cadence
- every 30 min
- stale at
- 6 h
- dedupe
HN item id- feeds
detectors — research
cloud
Subreddit and search feeds, while they last: Reddit ends RSS feeds on Nov 13, 2026 and closes public API access in March 2027 (PPC Land), so plan a replacement route.
- sign-in
- none (feeds)
- cadence
- hourly
- stale at
- 12 h
- dedupe
post id- feeds
detectors — research
cloud
Product Hunt
The public launch feed (its API's terms rule out commercial use).
- sign-in
- none
- cadence
- daily
- stale at
- 24 h
- dedupe
post id- feeds
detectors
cloud
TrustMRR
Poll the listings: startups with verified revenue, so launches and paying products.
- sign-in
- none
- cadence
- daily
- stale at
- 48 h
- dedupe
listing url- feeds
detectors — research
cloud
GitHub
Releases and issues on repos you follow, through the API.
- sign-in
- token
- cadence
- daily
- stale at
- 72 h
- dedupe
release or issue id- feeds
detectors — research
Competitors and reviews — what competitors ship and what their customers say
cloud
Competitor changelogs
Fetch each competitor's changelog page or feed; a new entry or a content-hash change is an item.
- sign-in
- none
- cadence
- daily
- stale at
- 72 h
- dedupe
entry url plus hash- feeds
detectors — research
cloud
Review sites and app stores
G2, Capterra, Trustpilot, the App Store and Google Play: newest reviews per product page.
- sign-in
- varies by site
- cadence
- daily
- stale at
- 72 h
- dedupe
review id or url- feeds
detectors — research
Your saves — bookmarks and likes: the strongest label you have
cloud
X bookmarks
The X API with the user's sign-in (bookmark read scope), folder by folder.
- sign-in
- OAuth
- cadence
- daily
- stale at
- 7 days
- dedupe
tweet id- feeds
labels — drafting · research
cloud
X handles and lists
Accounts you choose to follow closely, read through the API.
- sign-in
- paid API
- cadence
- hourly
- stale at
- 24 h
- dedupe
tweet id- feeds
detectors — drafting
cloud
YouTube liked videos
The API's liked list with the user's sign-in. Watch Later isn't exposed by the API.
- sign-in
- OAuth
- cadence
- daily
- stale at
- 7 days
- dedupe
video id- feeds
labels — drafting
import
your machine
Instagram saves
No API for saved posts: the account's data download, a share-to link, or an extension in the user's own browser.
- sign-in
- none server-side
- cadence
- weekly import
- stale at
- none
- dedupe
post shortcode- feeds
labels — drafting
import
your machine
Threads saves
No API for saves: share-to link, or the data download where it includes them.
- sign-in
- none server-side
- cadence
- weekly import
- stale at
- none
- dedupe
post id- feeds
labels — drafting
import
TikTok favorites
No API for favorites: TikTok's data download, or a share link per video.
- sign-in
- none server-side
- cadence
- on upload
- stale at
- none
- dedupe
video id- feeds
labels — drafting
You send it — links pushed in by hand
push
Links you send
A share sheet, an email-to-save address, a save command in chat, or an AI assistant connector.
- sign-in
- none
- cadence
- on arrival
- stale at
- 30 days
- dedupe
normalized url- feeds
research — labels
health
Three numbers say whether a source is working.
The newest item's age against the source's max staleness, the last run's status, and the runs in the last 24 hours by status.
ok — Newest item inside the max staleness, and the last run succeeded.
degraded — Data still fresh, but runs are partial or erroring. Some feeds in the source are failing.
stalled — No new item for longer than the max staleness. Either it broke or it was switched off; the run log says which.
elsewhere — Owned by a different runner (another machine). This runner can't see its run log, only its items.
| Source | Configured | Items | Newest item | Runs, 24h | Max stale | Health |
|---|---|---|---|---|---|---|
| Reddit search | 12 subreddits | 2,140 | Mar 2, 08:50 | 24 ok | 12 h | ok |
| Hacker News | search | 980 | Mar 2, 08:30 | 48 ok | 6 h | ok |
| RSS | 14 feeds | 3,218 | Mar 2, 08:41 | 22 ok · 2 partial | 24 h | degraded |
| Launch directory | daily feed | 310 | Mar 2, 00:10 | 4 ok | 48 h | ok |
| Newsletter inbox | 3 lists | 75 | Feb 28, 14:00 | 2 ok | 96 h | ok |
| YouTube | 6 channels | 120 | Mar 1, 18:20 | run elsewhere | 96 h | elsewhere |
| Review sites | 4 competitors | 90 | Feb 11 | none | 168 h | stalled |
Illustrative values, invented to show the shape.
reading it
In this illustrative reading, RSS is the one to look at: two runs in the last day were partial, so at least one feed is failing. Review sites read stalled because collection was paused on purpose; a paused state with a reason keeps that from looking like an outage.
what the items feed
One store, four readers.
Items are written once. Each reader takes what it needs and writes its own derived rows, so adding a reader never means re-collecting.
reader 1 — Detectors · Any source can be the "where" of a detector. Its items get matched against the detector's rule and become signals. · Watch your newsletters for a competitor's pricing change; the email lands, the change detector fires.
reader 2 — Drafting · What you read and saved this week becomes topics and context for drafts, so posts track what's current. · Three saved threads on onboarding this week: tomorrow's drafts lean that way.
reader 3 — Research · Entity pages and dossiers built from items: every person, product and company mentioned. · A competitor's changelog, reviews and launch posts on one page, with dates.
reader 4 — Labels · A save is a strong positive label. It trains a ranker for what this founder finds valuable, so feeds sort by it. · Saved 40 of 2,000 items this month: those 40 teach the ranker; the rest are soft negatives.
privacy and terms
Six rules every connector follows.
Official routes first — Use the platform's API, feed or data export before anything else. Check each source's current terms before automating it.
Session-only means your machine — Anything visible only while signed in runs on the user's own machine, under their account, and reads only their own saves.
Private per account — Items belong to one account. They never train or feed anyone else's pipeline.
Delete on disconnect — Disconnecting a source deletes its items and derived rows, not just the connector.
Sign-ins stay with the runner — Tokens live where they're used: cloud tokens in an encrypted vault, device sessions on the device. Only a reference is stored on the source.
Public data, read gently — Respect rate limits and each community's rules. Never collect from member-only spaces without the admins' say.
how it connects
Upstream of detectors.
Detector catalog — NEXT · STAGE 5 · Its source directory lists the public places detectors read; this page adds the founder's own.
Buying Moments Map — STAGE 4 · The triggers a source's items can reveal.
GTM tracking plan — STAGE 11 AND 12 · Run and item events join the same event log.
Pipeline — THE MAP · All 12 stages on one canvas.
data to capture
What sources record, and what it trains.
The same four lines every stage in this guide uses: record, label, trains, test.
record
- every run: started, finished, status, items seen, new and updated, cursor or ETag, error, cost
- every item raw, with its url, published time, captured time and a content hash
- deletions at the source (the item stays, marked gone, with the time)
- which detectors read each item, and which matched it
- saves and unsaves with their time, and which items later went into a draft or reply
label
- saved, so valued; unsaved soon after, so not
- used in a draft, a reply or research: valued
- matched and marked real by the founder: a good source for that trigger
- disconnected: the founder didn't value the source
trains
- a per-founder ranker: what to read first
- source value: signals, leads, drafts and payments per source, per dollar
- expected cadence per source, so staleness alerts fire on the source's own rhythm
test
- cadence and max staleness per kind · A/B by source, measured on alerts versus real outages
- adding a source to a detector · shadow: run it beside live on the same trigger and compare matches
- the ranker · offline replay on past saves