Add unit tests, security hardening, and error logging improvements#43
Open
huangyingting wants to merge 1 commit into
Open
Add unit tests, security hardening, and error logging improvements#43huangyingting wants to merge 1 commit into
huangyingting wants to merge 1 commit into
Conversation
- Add test suite using Node.js built-in test runner (test/utils.test.mjs, test/fetch-url.test.mjs) covering utils functions, URL handling, caching, bot detection, HTML extraction, and more. - Add 'npm run test' script to root package.json. - Security: reject non-http(s) URL schemes in canonicalSourceUrl() to prevent javascript:/data:/file: injection. - Security: add URL scheme validation in fetchUrl() to block SSRF via disallowed protocols. - Security: enforce MAX_BODY_BYTES (50 MB) on fetch responses to prevent memory exhaustion from oversized downloads. - Error handling: replace silent catch blocks with console.warn() logging in check-reports.mjs (folderDigest, loadCache), utils.mjs (listDirs), and fetch.mjs (readCache, loadHostStrategies) so failures are diagnosable. - Add .nvmrc pinning Node 22. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
High-priority improvements identified during a full project review:
1. Unit Tests (new)
node:testrunner — no new dependencies neededtest/utils.test.mjs— coversslugify,normalizeCompanyName,normalizeDomain,canonicalSourceUrl,companySlugFromRunId,runDateFromRunId,isRunId,researchCacheDir,hasText,parseDate,registrableDomain,collectClaimRefs,normalizeRevisiontest/fetch-url.test.mjs— coverscanonicalCacheKey,looksLikePdfBuffer,looksLikeBotChallenge,waybackUrl,readerUrl,htmlToText,cleanExtractedText,extractTitle,stripWaybackToolbar,registrableDomainnpm run testto rootpackage.json2. Security Hardening
canonicalSourceUrl()now rejectsjavascript:,data:,file:and other non-http(s) schemes (returns'')fetchUrl()validates URL scheme before making any network requestfetchUrl()enforces a 50 MB max response body size (MAX_BODY_BYTES) — checks bothContent-Lengthheader and actual body3. Error Handling
catch {}→return null/falsepatterns withconsole.warn()logging so failures are diagnosable:check-reports.mjs:folderDigest()andloadCache()now log the specific errorutils.mjs:listDirs()logs which path failed to statfetch.mjs:readCache()andloadHostStrategies()log cache read failures4. DX
.nvmrcpinning Node 22 for consistent environmentsTesting
Run:
npm run test