listen · your own sources
Source health
Freshness, runs and stalled sources.
In Your own sources
A source that broke and a source that had a quiet week look exactly the same from the outside: no new items. Source health tells them apart with three numbers per source: how old the newest item is against its max staleness, how the last run ended, and how the runs in the last 24 hours went.
runs and items → health check · per source → state · ok, degraded, stalled, elsewhere → alert or quiet
rule 1 — Freshness has a budget · Every source has a max staleness in hours. The newest item older than that is a problem, whatever the run log says.
rule 2 — Partial is not success · A run where some feeds failed counts against the source, even when the others brought items.
rule 3 — Off on purpose is its own state · A source switched off deliberately should read paused, not stalled, so real outages stand out.
what it is
Data freshness monitoring, per source.
In data-engineering terms this is a freshness SLO plus job monitoring: each source promises new data within a window, each run reports its status, and the two combine into one health state.
Max staleness — freshness SLO · Hours a source may go without a new item.
Last run status — job status · success, partial or error for the most recent fetch.
Runs in 24h — run counts · success, partial and error counts over the last day. One partial run is a blip; 24 is a broken feed.
Owned here — runner ownership · Whether this runner owns the source. If not, it can see the items but not the run log.
Health — derived state · ok, degraded, stalled, or not owned here (shown as elsewhere).
Expected cadence — learned rhythm · How often this source normally posts, so a quiet but healthy source isn't flagged.
how it works
One decision per source, every time the check runs.
The check runs on its own schedule, separate from the fetches, so a dead scheduler still shows up.
- Read the source row — configured · owner
- Owned here? — no → elsewhere
- Newest item age — vs max staleness
- Older? — yes → stalled
- Last run + 24h runs — partial or error?
- Any? — yes → degraded
- Otherwise — ok
- Snapshot — store and alert
Who owns it
Is the data fresh
Compare now minus the newest item's time with max staleness. Past it, the source is stalled, whatever the runs say.
Are the runs clean
If the last run was partial or error, or any run in the last 24 hours was, the source is degraded: data is arriving, but something inside it is failing.
Otherwise ok
Fresh data and clean runs.
Store a snapshot, alert on changes
Write the state with its inputs every check. Alert when a source moves into stalled or degraded, not every time it stays there.
the object
The health row, field for field.
One row per source, recomputed on every check. The sample below is invented.
| Field | Type | What it holds |
|---|---|---|
source_type | text | reddit, hn, rss, youtube, … |
collector | text | Which fetcher reads it |
configured | int | null | How many feeds, handles or repos are set up |
posts | int | Items stored so far |
newest_post · oldest_post | timestamp | Freshness and history depth |
last_fetched | timestamp | When a run last brought something back |
last_run_status · last_run_at | enum · timestamp | success, partial, error, running, or none |
runs_24h | {success, partial, error} | Counts over the last day |
runner | text | Which job or worker is meant to run it |
owned_here | bool | Whether this runner owns it |
max_staleness_h | int | The freshness budget |
health | enum | ok, degraded, stalled, not_owned_here |
{
"source_type": "rss",
"collector": "rss",
"configured": 14,
"posts": 3218,
"newest_post": "2026-03-02T08:41:00+00:00",
"oldest_post": "2025-11-19T16:05:00+00:00",
"last_fetched": "2026-03-02T09:00:12Z",
"last_run_status": "success",
"last_run_at": "2026-03-02T09:00:04Z",
"runs_24h": {
"success": 22,
"partial": 2,
"error": 0
},
"runner": "worker-1",
"owned_here": true,
"max_staleness_h": 24,
"health": "degraded"
}Seven sources, four states, one morning (illustrative).
Runs in 24h read success · partial · error. Times are UTC.
| Source | Configured | Items | Newest item | Runs, 24h | Max stale | Health |
|---|---|---|---|---|---|---|
| Reddit search | 12 subreddits | 2,140 | Mar 2, 08:50 | 24 ok | 12 h | ok |
| Hacker News | search | 980 | Mar 2, 08:30 | 48 ok | 6 h | ok |
| RSS | 14 feeds | 3,218 | Mar 2, 08:41 | 22 ok · 2 partial | 24 h | degraded |
| Launch directory | daily feed | 310 | Mar 2, 00:10 | 4 ok | 48 h | ok |
| Newsletter inbox | 3 lists | 75 | Feb 28, 14:00 | 2 ok | 96 h | ok |
| YouTube | 6 channels | 120 | Mar 1, 18:20 | run elsewhere | 96 h | elsewhere |
| Review sites | 4 competitors | 90 | Feb 11 | none | 168 h | stalled |
Illustrative values, invented to show the shape.
reading it
Illustrative reading: RSS needs a look, since two runs in the last day were partial, so at least one feed is failing. Review sites read stalled because collection was paused on purpose; a paused state with a reason would stop that reading like an outage.
failure modes
How health checks themselves fail.
Quiet looks broken — A low-volume source (a manual link list) trips stalled every week. · catch itLearn each source's expected cadence; alert on gaps longer than its own rhythm.
Partial hides inside totals — Twenty feeds work, two fail forever; the source looks fine on item counts. · catch itPer-feed status inside the run, and degraded on any partial run.
Off on purpose reads as an outage — A source switched off on purpose shows up as stalled. · catch itA paused state with a reason and a date.
The checker died — No alerts because the health job itself stopped. · catch itRun the check on its own schedule and alert on a missing snapshot.
Time zones — A newest-item time stored without a zone looks hours stale. · catch itStore every timestamp in UTC with the offset.
data to capture
What source health records.
The same four lines every stage uses, under the guide's eight data rules: log the decision, version what decided, freeze the inputs, keep the negatives, log the odds, every human touch is a label, one chain of ids, keep the raw.
record
- a health snapshot per source per check, with all its inputs
- state changes with the time they happened
- alerts sent and whether anyone acted on them
- paused periods, with who paused and why
label
- an alert confirmed as a real outage, or dismissed as a false alarm
- the fix applied (feed url changed, sign-in renewed, route replaced)
trains
- expected cadence per source
- an anomaly check that beats a fixed max staleness
- time-to-detect and time-to-fix per kind
test
- fixed vs learned staleness thresholds · offline replay on past snapshots, scored on true outages caught vs false alarms