Add wiki-bench v0 fixture-only retrieval benchmark - #845
Open
tobi wants to merge 3 commits into
Open
Conversation
Ship queries + expected paths only (no private wiki page bodies). Includes schema validation that always runs in CI, and optional BM25 floors that skip unless QMD_WIKI_BENCH_DOCS or QMD_WIKI_PATH points at a local corpus.
|
Dear tobi/qmd,
We would like to acknowledge that we have received your request and a ticket has been created.
A support representative will be reviewing your request and will send you a personal response.(usually within 24-48 hours).
Thank you for your patience.
…On Thu, Aug 13 2026, at 12:35 AM, tobi/qmd ***@***.***> wrote:
Summary Fixture-only wiki-bench v0 for retrieval quality over the public-appearances wiki. This deliberately does not vendor any private wiki page bodies (unlike closed PR #844, which was deleted for that reason). (#844)
* Ships src/bench/fixtures/wiki-v0.json (44 queries; expected_files are wiki-relative paths only)
* Always-on schema test: test/wiki-bench-fixture.test.ts
* Optional BM25 quality floors: test/wiki-bench-bm25.test.ts — skips unless QMD_WIKI_BENCH_DOCS or QMD_WIKI_PATH points at a local corpus directory
* Docs: docs/WIKI_BENCH.md
No test/wiki-bench-docs/**, no sync script that writes corpus into the tree, no wiki .md bodies anywhere in this PR.
Privacy Included Not included Fixture JSON (queries + expected paths/ids) Private tobi/wiki page bodies Schema + env-gated BM25 tests test/wiki-bench-docs/** docs/WIKI_BENCH.md Any sync that vendors corpus into the repo
Default CI (no corpus env) stays green.
How to validate npx vitest run test/wiki-bench-fixture.test.ts # Always (no corpus needed) # Optional local BM25 floors export QMD_WIKI_BENCH_DOCS=/path/to/wiki/wiki npx vitest run test/wiki-bench-bm25.test.ts # or QMD_WIKI_PATH qmd collection add /path/to/wiki/wiki --name wiki-bench
qmd bench src/bench/fixtures/wiki-v0.json -c wiki-bench # Full multi-backend harness after indexing a local checkout
See docs/WIKI_BENCH.md for details. (https://github.com/tobi/qmd/blob/wiki-bench-v0-fixture/docs/WIKI_BENCH.md)
Test plan npx vitest run test/wiki-bench-fixture.test.ts (green, no wiki docs) npx vitest run test/wiki-bench-bm25.test.ts (4 skipped + gate pass when env unset) Confirmed commit has only: fixture JSON, two test files, docs/WIKI_BENCH.md CI on this PR You can view, comment on, or merge this pull request online at: #845 (#845) Commit Summary e335fbf Add wiki-bench v0 fixture-only retrieval benchmark (e335fbf) File Changes
( 4 files) (https://github.com/tobi/qmd/pull/845/files) A docs/WIKI_BENCH.md
(95) (https://github.com/tobi/qmd/pull/845/files#diff-4d301bfb7213849b16efc4f1acd2419e788cb3139d8d58e16fdbf52e041f8123) A src/bench/fixtures/wiki-v0.json
(365) (https://github.com/tobi/qmd/pull/845/files#diff-200bd5aa82cf9663e01a517d908d0ee915f0394ff379e6eb8e19b2c34076f81c) A test/wiki-bench-bm25.test.ts
(222) (https://github.com/tobi/qmd/pull/845/files#diff-f630b98bef35a592a7e0e3fd0469095ce72d8ec74aeb9bece2f1bd4e37a2c66c) A test/wiki-bench-fixture.test.ts
(84) (https://github.com/tobi/qmd/pull/845/files#diff-87f2fc09455a13fe8eb93150f2c801f35cd9d021742a7d7624b198adf6f5cf72) Patch Links: https://github.com/tobi/qmd/pull/845.patch (https://github.com/tobi/qmd/pull/845.patch) https://github.com/tobi/qmd/pull/845.diff (https://github.com/tobi/qmd/pull/845.diff)
—
Reply to this email directly, view it on GitHub, or (#845?email_source=notifications&email_token=BRF3HG4INWSOYRZCKFW76LT5JUEOPA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42DENRXGA3TQMRWGKTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVRTG633UMVZF6Y3MNFRWW) unsubscribe. (https://github.com/notifications/unsubscribe-auth/BRF3HG7SXTDL727A3BC6CZL5JUEOPAVCNFSNUABGKJSXA33TNF2G64TZHMYTCMJSGM3DKMZQGE5US43TOVSTWNJRGM3DKNZQHAZTJILWAI)
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and (https://github.com/notifications/mobile/ios/BRF3HGYEFPHI5E7ZSSZT3TT5JUEOPA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42DENRXGA3TQMRWGKTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVJTG633UMVZF62LPOM) Android. Download it today! (https://github.com/notifications/mobile/android/BRF3HG7563QV2DSNJHNVWCL5JUEOPA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42DENRXGA3TQMRWGKTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVZTG633UMVZF6YLOMRZG62LE)
You are receiving this because you are subscribed to this thread. Message ID: <tobi/qmd/pull/845 @ github . com> [
{
***@***.***": "http://schema.org",
***@***.***": "EmailMessage",
"potentialAction": {
***@***.***": "ViewAction",
"target": "#845?email_source=notifications\u0026email_token=BRF3HG45WPEBJBHUJ4BIIBL5JUEOPA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42DENRXGA3TQMRWGKTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVNTW2YLJNRPWG3DJMNVQ",
"url": "#845?email_source=notifications\u0026email_token=BRF3HG45WPEBJBHUJ4BIIBL5JUEOPA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42DENRXGA3TQMRWGKTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVNTW2YLJNRPWG3DJMNVQ",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
***@***.***": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
|
Indexes a temporary wiki-bench collection outside the git tree from QMD_WIKI_PATH/wiki or QMD_WIKI_BENCH_DOCS, runs BM25 via runBenchmark, and fails if exact/alias quality floors are not met. No wiki page bodies are committed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixture-only wiki-bench v0 for retrieval quality over the public-appearances wiki. This deliberately does not vendor any private wiki page bodies (unlike closed PR #844, which was deleted for that reason).
src/bench/fixtures/wiki-v0.json(44 queries;expected_filesare wiki-relative paths only)test/wiki-bench-fixture.test.tstest/wiki-bench-bm25.test.ts— skips unlessQMD_WIKI_BENCH_DOCSorQMD_WIKI_PATHpoints at a local corpus directorydocs/WIKI_BENCH.mdNo
test/wiki-bench-docs/**, no sync script that writes corpus into the tree, no wiki.mdbodies anywhere in this PR.Privacy
tobi/wikipage bodiestest/wiki-bench-docs/**docs/WIKI_BENCH.mdDefault CI (no corpus env) stays green.
How to validate
See docs/WIKI_BENCH.md for details.
Test plan
npx vitest run test/wiki-bench-fixture.test.ts(green, no wiki docs)npx vitest run test/wiki-bench-bm25.test.ts(4 skipped + gate pass when env unset)docs/WIKI_BENCH.md