Skip to content

docs: scope the v0.4.0 enrichment feeds (EPSS · CISA KEV · endoflife.date) - #213

Open
tyler-rich wants to merge 2 commits into
devfrom
claude/v0.4.0-enrichment-scoping-epwqaz
Open

docs: scope the v0.4.0 enrichment feeds (EPSS · CISA KEV · endoflife.date)#213
tyler-rich wants to merge 2 commits into
devfrom
claude/v0.4.0-enrichment-scoping-epwqaz

Conversation

@tyler-rich

@tyler-rich tyler-rich commented Aug 14, 2026

Copy link
Copy Markdown
Owner

Adds docs/scoping/v0.4.0-enrichment.md, fixing the shape of the Track B Phase 1 enrichment work before any of it is built. Documentation only — no code, schema, migration, endpoint, or configuration changes. The document proposes four tables and a config surface; it does not implement them.

The three feeds read as three features but share one mechanism — scheduled sync of a public feed into a local cache, read at display time, offline-graceful. Building that three times is what this document exists to prevent.

The key-space call

Share the mechanism; do not share the key space.

The deciding argument isn't the key type — it's that the feeds don't enrich the same object. EPSS and KEV enrich a finding; endoflife.date enriches a target, so a 400-finding scan of an EOL base image has one EOL fact, not 400. Routing it through a findings-level join would broadcast one fact across every row that happens to share a scan.

The key spaces confirm it independently: findings.vuln_id already carries the CVE ID verbatim, so EPSS and KEV join by equality on a value that exists. endoflife.date has no key in Scrye at all — one must be manufactured from scan metadata, matched by version prefix against a release cycle, then compared to today's date. Upstream's own identifiers won't close that gap either: products/python.md declares purl: entries while products/debian.md declares cpe:.

Result: one FeedSource protocol + sync engine + scheduler hook + import seam; one polymorphic feed_sync_state table; three typed cache tables; two read paths. A single polymorphic cache table is rejected on four grounds, the sharpest being that it breaks EPSS sorting.

Verified at source

KEV and endoflife.date were verified against genuinely upstream sources (cisagov/kev-data is CISA-operated; endoflife-date/endoflife.date is the dataset the API is generated from) — the real 1,665-entry catalogue, its draft-07 schema, cadence, and CC0 licensing; endoflife.date's MIT LICENSE, its v0 OpenAPI schema, and its Beta-API warning.

Three of those findings changed the design:

  • endoflife.date is MIT, but product descriptions are CC BY-SA 3.0 (Wikipedia-derived) — share-alike, which MIT isn't. Recommendation: don't ingest that field; the feature needs dates and cycles, not prose.
  • KEV's dueDate is populated on every row, but only one entry is still in the future — the roadmap's "remediation countdown" is an overdue badge in the common case.
  • eoas / eol / eoes are three distinct dates. Debian 12 is past eol but inside eoes — and it's the base of Scrye's own runtime image, so a naive "past eol ⇒ red" flag fires on our own supported base.

Claims carry one of four tags — [VERIFIED], [MEASURED], [ASSERTED], [UNVERIFIED] — so a later session can tell what was checked from what was reasoned.

The EPSS section is deliberately unverified. Every EPSS host was unreachable from the authoring environment, and the section was left as a work order — required facts plus a runnable curl script — rather than filled in from prior knowledge or secondary sources. A follow-up session completes it. Nothing in §§ 5–13 depends on an EPSS fact beyond a row-count sizing premise that's called out where it's used.

Measured, not estimated

  • epss_scores WITHOUT ROWID: 10.23 MiB vs 17.68 MiB at 280k rows, and faster to look up
  • sort-by-EPSS LEFT JOIN: 3.5 ms for a 100-row page over 500k findings × 280k scores
  • atomic cache swap: 0.17 s write-lock window against the configured 5,000 ms busy_timeout

The join plan resolves the driving side by the existing ix_findings_scan_severity_vuln, so no new index on findings is needed — which is what keeps the whole design additive. Benchmarks ran on SQLite 3.45.1, not the runtime image; the document says so.

Two consequences worth flagging

  • Backup silently includes new tables. bundle.py builds from Base.metadata.sorted_tables minus an exclusion list, and encrypts in memory in a single pass (existing warning at 250k findings rows). All four new tables — feed_sync_state included — need to join _EXCLUDED_TABLES.
  • The egress guard needs no change. All three feeds are public hosts and pass validate_egress_url() as-is. SCRYE_ALLOW_INTERNAL_EGRESS is not an enrichment prerequisite — it only widens the guard to RFC-1918, and is correct in exactly one case: a feed URL overridden to an internal mirror. The real gap is redirects, since the guard validates only the host you asked for.

Open stop-and-ask

EOL needs a product identity for a scan, which Scrye doesn't store. The document recommends a scan_platform side table populated from Trivy's Metadata.OS at normalize time (reading the raw artifact at read time breaks under retention). That's a data-model addition, so it's recorded as recommended, not approved — § 14 item 2 requires confirmation before it's built.

See docs/ARCHIVE.md § 14 (2026-08-14) for the decisions and reasoning recorded for this work.

…e.date)

Fix the shape of the enrichment mechanism before any of it is built, so the
three Track B Phase 1 feeds are built once rather than three times.

Key call: one sync framework and one shared feed_sync_state table, but three
typed cache tables and two read paths. EPSS and KEV enrich a finding and share
a CVE key that findings.vuln_id already carries; endoflife.date enriches a
target, has no key in Scrye at all, and needs version-prefix matching plus a
date comparison. Unifying the key space would broadcast one fact across every
finding in a scan rather than join to it.

KEV and endoflife.date are verified at upstream sources; every EPSS fact is
marked UNVERIFIED with the command to run, because all EPSS hosts were
unreachable from the authoring session.

Additive only: no change to the findings table, no locked decision touched, and
no scan path that depends on a network fetch.
Logs the key-space call and its reasoning, the source-verification results for
KEV and endoflife.date, the three verified findings that changed the design
(the CC BY-SA 3.0 product-description field, KEV dueDate being an overdue badge
rather than a countdown, and eoas/eol/eoes being three dates), the four
benchmarked figures, the backup-exclusion and egress-guard consequences, and
the open scan_platform stop-and-ask.

Also records that the EPSS section landed unverified by maintainer direction
after the authoring session's egress policy blocked every EPSS host, so a later
reader reads it as a work order rather than as an omission.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant