Skip to content

adopt CDC scraper with full-text HTML extraction - #35

Closed
NaitikOza wants to merge 1 commit into
MedARC-AI:mainfrom
NaitikOza:cdc-scraper-update
Closed

NaitikOza wants to merge 1 commit into
MedARC-AI:mainfrom
NaitikOza:cdc-scraper-update

Conversation

@NaitikOza

Copy link
Copy Markdown

Summary

Adopts the unfinished CDC implementation from #7 on the current shared scraping registry and implements the maintainer's request to use CDC's HTML instead of Selenium-driven PDF downloads and GROBID.

  • Discover records from the CDC Stacks Guidelines and Recommendations collection using HTTP + lxml.
  • Select English, public-domain records with an official CDC MMWR full-text HTML DOI.
  • Follow the DOI to cdc.gov and convert only the semantic syndicate article body to markdown.
  • Skip PDF-only, translated, restricted, and external-publisher records instead of treating Stacks abstracts as full guidelines.
  • Preserve Stacks/DOI provenance, authors, corporate authors, publication type/year, license, and content_scope: full_html.
  • Register --source cdc, support explicit Stacks record URLs, document usage, and honor Stacks' 10-second listing crawl delay.
  • Use an identifiable AMFV user agent because Stacks rejects the shared synthetic Chrome user agent.

No browser, Selenium, GROBID, OCR, or new dependency is required.

Scope and provenance

The original #7 collection target and record-discovery approach are credited in the module header. The implementation commit includes Co-authored-by: azka2001 <azkaah2001@gmail.com>. The original PR remains open for its review history.

The content boundary is deliberately conservative. CDC Stacks currently mixes complete CDC publications, PDF-only records, translations, and third-party publications. This scraper emits a document only when it can verify and extract an official full-text CDC HTML body.

Validation

  • uv run ruff check .
  • uv run ruff format --check .
  • uv run pytest -q — 61 passed
  • uv build --package amfv-datasets
  • Live listing smoke test: uv run amfv-scrape --source cdc --documents 1 --no-progress
  • Live URL smoke test: Stacks record cdc:182510 resolved to the official MMWR article and produced 49,013 characters across 8 sections with full provenance

Co-authored-by: azka2001 <azkaah2001@gmail.com>
@NaitikOza NaitikOza closed this Oct 25, 2026
@NaitikOza
NaitikOza deleted the cdc-scraper-update branch October 25, 2026 01:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant