docs: scope the v0.4.0 enrichment feeds (EPSS · CISA KEV · endoflife.date) - #213
Open
tyler-rich wants to merge 2 commits into
Open
docs: scope the v0.4.0 enrichment feeds (EPSS · CISA KEV · endoflife.date)#213tyler-rich wants to merge 2 commits into
tyler-rich wants to merge 2 commits into
Conversation
…e.date) Fix the shape of the enrichment mechanism before any of it is built, so the three Track B Phase 1 feeds are built once rather than three times. Key call: one sync framework and one shared feed_sync_state table, but three typed cache tables and two read paths. EPSS and KEV enrich a finding and share a CVE key that findings.vuln_id already carries; endoflife.date enriches a target, has no key in Scrye at all, and needs version-prefix matching plus a date comparison. Unifying the key space would broadcast one fact across every finding in a scan rather than join to it. KEV and endoflife.date are verified at upstream sources; every EPSS fact is marked UNVERIFIED with the command to run, because all EPSS hosts were unreachable from the authoring session. Additive only: no change to the findings table, no locked decision touched, and no scan path that depends on a network fetch.
Logs the key-space call and its reasoning, the source-verification results for KEV and endoflife.date, the three verified findings that changed the design (the CC BY-SA 3.0 product-description field, KEV dueDate being an overdue badge rather than a countdown, and eoas/eol/eoes being three dates), the four benchmarked figures, the backup-exclusion and egress-guard consequences, and the open scan_platform stop-and-ask. Also records that the EPSS section landed unverified by maintainer direction after the authoring session's egress policy blocked every EPSS host, so a later reader reads it as a work order rather than as an omission.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
docs/scoping/v0.4.0-enrichment.md, fixing the shape of the Track B Phase 1 enrichment work before any of it is built. Documentation only — no code, schema, migration, endpoint, or configuration changes. The document proposes four tables and a config surface; it does not implement them.The three feeds read as three features but share one mechanism — scheduled sync of a public feed into a local cache, read at display time, offline-graceful. Building that three times is what this document exists to prevent.
The key-space call
Share the mechanism; do not share the key space.
The deciding argument isn't the key type — it's that the feeds don't enrich the same object. EPSS and KEV enrich a finding; endoflife.date enriches a target, so a 400-finding scan of an EOL base image has one EOL fact, not 400. Routing it through a findings-level join would broadcast one fact across every row that happens to share a scan.
The key spaces confirm it independently:
findings.vuln_idalready carries the CVE ID verbatim, so EPSS and KEV join by equality on a value that exists. endoflife.date has no key in Scrye at all — one must be manufactured from scan metadata, matched by version prefix against a release cycle, then compared to today's date. Upstream's own identifiers won't close that gap either:products/python.mddeclarespurl:entries whileproducts/debian.mddeclarescpe:.Result: one
FeedSourceprotocol + sync engine + scheduler hook + import seam; one polymorphicfeed_sync_statetable; three typed cache tables; two read paths. A single polymorphic cache table is rejected on four grounds, the sharpest being that it breaks EPSS sorting.Verified at source
KEV and endoflife.date were verified against genuinely upstream sources (
cisagov/kev-datais CISA-operated;endoflife-date/endoflife.dateis the dataset the API is generated from) — the real 1,665-entry catalogue, its draft-07 schema, cadence, and CC0 licensing; endoflife.date's MITLICENSE, its v0 OpenAPI schema, and its Beta-API warning.Three of those findings changed the design:
dueDateis populated on every row, but only one entry is still in the future — the roadmap's "remediation countdown" is an overdue badge in the common case.eoas/eol/eoesare three distinct dates. Debian 12 is pasteolbut insideeoes— and it's the base of Scrye's own runtime image, so a naive "past eol ⇒ red" flag fires on our own supported base.Claims carry one of four tags —
[VERIFIED],[MEASURED],[ASSERTED],[UNVERIFIED]— so a later session can tell what was checked from what was reasoned.The EPSS section is deliberately unverified. Every EPSS host was unreachable from the authoring environment, and the section was left as a work order — required facts plus a runnable
curlscript — rather than filled in from prior knowledge or secondary sources. A follow-up session completes it. Nothing in §§ 5–13 depends on an EPSS fact beyond a row-count sizing premise that's called out where it's used.Measured, not estimated
epss_scoresWITHOUT ROWID: 10.23 MiB vs 17.68 MiB at 280k rows, and faster to look upLEFT JOIN: 3.5 ms for a 100-row page over 500k findings × 280k scoresbusy_timeoutThe join plan resolves the driving side by the existing
ix_findings_scan_severity_vuln, so no new index onfindingsis needed — which is what keeps the whole design additive. Benchmarks ran on SQLite 3.45.1, not the runtime image; the document says so.Two consequences worth flagging
bundle.pybuilds fromBase.metadata.sorted_tablesminus an exclusion list, and encrypts in memory in a single pass (existing warning at 250k findings rows). All four new tables —feed_sync_stateincluded — need to join_EXCLUDED_TABLES.validate_egress_url()as-is.SCRYE_ALLOW_INTERNAL_EGRESSis not an enrichment prerequisite — it only widens the guard to RFC-1918, and is correct in exactly one case: a feed URL overridden to an internal mirror. The real gap is redirects, since the guard validates only the host you asked for.Open stop-and-ask
EOL needs a product identity for a scan, which Scrye doesn't store. The document recommends a
scan_platformside table populated from Trivy'sMetadata.OSat normalize time (reading the raw artifact at read time breaks under retention). That's a data-model addition, so it's recorded as recommended, not approved — § 14 item 2 requires confirmation before it's built.See
docs/ARCHIVE.md§ 14 (2026-08-14) for the decisions and reasoning recorded for this work.