Multi-paper implementation spec synthesizer. Feed it a set of related papers; get back a single reconciled, citation-traced, contradiction-annotated implementation specification that a human approves before any code is written.
PaperCoder converts one paper into code. PaperSynth converts many papers into one verified spec that a human approves before any code exists.
Implementing a method from the literature fails in four recurring ways:
| Failure mode | Consequence |
|---|---|
| Equations and algorithms degrade during PDF extraction | Silent numerical errors |
| Papers omit implementation details | The coding agent quietly guesses |
| Related papers prescribe conflicting methods | The implementer picks arbitrarily, or blends incompatibly |
| No traceability from code back to a paper section | Reproduction failures are impossible to audit |
PaperSynth addresses all four by making provenance mandatory, contradictions first-class, and the human review gate non-optional.
It deliberately stops before code generation. A wrong hyperparameter caught in a 200-line YAML costs minutes; the same error caught after a failed 12-hour training run costs a day and gets misattributed to a bug elsewhere.
ingest → extract → verify → align → contradict → reconcile → gapcheck → synthesize
│ │ │ │ │ │ │ │
per-paper ─┴─────────┘ └───────── corpus-wide ┴───────────┴──────────┘
Stages 0–2 run per paper and are parallelizable. Stages 3–6 reason across the corpus. Stage 7 emits implementation_spec.yaml — the deliverable.
Every stage writes a human-readable artifact to runs/{run_id}/, so any decision can be inspected after the fact.
These are the non-negotiable ones. The rest is implementation detail.
- Every claim carries provenance. A claim whose payload cannot be located in its cited span is rejected, not downgraded. A spec field with no traceable source blocks emission.
- Extractors never see another paper. If extraction were corpus-aware, an LLM holding Paper A's learning rate in context while extracting Paper B's would drift toward agreement — genuine contradictions would vanish before the detector ever ran.
- The reconciliation fallback is always ESCALATED. No policy rule fires unambiguously? A human decides. There is no "best guess" path. A tool that resolves 100% of conflicts is indistinguishable from one that resolves 60% correctly and 40% arbitrarily, unless it tells you which is which.
- Values absent from the paper are never invented. Absent means a
Gapgets raised, not a plausible default filled in. - IDs are content-derived. Re-ingesting the same PDF produces the same span IDs, so a spec emitted today still diffs cleanly against one from last month.
Pre-alpha, under active construction. Building the minimal viable architecture first, in dependency order:
- Schemas, contracts, and validator
- Span addressing with lossless round-trip
- Ingest (LaTeX + GROBID/PDF + arXiv)
- LLM layer with free-tier fallback chain
- Hyperparameter extractor
- Citation trace + range check
- Align + VALUE_CONFLICT detection
- Policy engine with ESCALATED fallback
- Spec builder + provenance closure gate
- CLI wiring for the full pipeline
- Gap check (
missing_but_critical) — static checklist pass - Remaining extractors: equation, algorithm
- SplitterAgent (opt-in; see below)
-
methodclaim type + METHOD_CONFLICT detection - Adversarial gap pass (
--adversarial-gaps) -
resultclaim type + RESULT_CONFLICT detection +expected_results - Symbol closure check
The MVA acceptance criterion now passes. Given three papers with one hand-planted conflicting hyperparameter, the pipeline emits a spec that (a) contains that conflict in open_conflicts with both positions and full provenance, and (b) contains no fabricated conflicts — the two values stated under different conditions are correctly left alone.
See tests/e2e/test_mva_acceptance.py. The end-to-end run is deterministic and offline: extraction responses are scripted, so the suite costs nothing and cannot drift.
The split gate (--split-gate) reviews clusters that span papers and rejects merges
of things that are only superficially similar. It works, and it is off by default
on the measurement rather than on principle.
On BERT/RoBERTa/ALBERT it correctly separated num_steps from warmup_steps and
hidden_dim into three model variants — but it also split batch_size into three
concepts, losing the genuine BERT-256 vs RoBERTa-8000 disagreement, which is that
corpus's headline finding. The splits it got right removed no false contradiction,
because condition-and-unit grouping had already kept those apart. One real finding
lost, none gained.
Enable it where its designed job actually arises — alongside --embedding-merges,
where it rejected all five bad merges that feature proposes:
papersynth run ... --embedding-merges # implies --split-gateRequires Python 3.11+.
git clone https://github.com/Daksha1611/papersynth.git
cd papersynth
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
papersynth --versionThe PDF path additionally needs GROBID:
docker run --rm -d --name grobid -p 8070:8070 lfoppiano/grobid:0.8.1
curl -s http://localhost:8070/api/isalive # expect: truepytest # unit + e2e + regressions, no network
ruff check . && ruff format --check .
mypy papersynthCheck the setup before spending anything — every one of these has failed at least once in development, each time surfacing later as a confusing error somewhere else:
papersynth doctorThen the review flow:
papersynth run --papers 1810.04805,1907.11692 --objective "..." --out runs/demo
papersynth conflicts runs/demo --status open # exits 2 while any remain
papersynth resolve runs/demo ctr_ce42684b --select clm_11f315 --note "why"
papersynth gaps runs/demo
papersynth spec runs/demo --format yaml
papersynth approve runs/demo --reviewer youresolve re-emits the spec from stage artifacts without re-extracting anything, so a
human decision costs no model calls. --resume reuses papers already extracted, for
picking a run back up after free-tier quotas reset.
Exit codes are part of the contract, so scripts can branch on them: 2 means a human
has to decide something, 3 from diff means a value moved that someone may already
have built against.
papersynth diff runs/demo runs/demo_v2 --format json
papersynth models --provider groq # free lineups rotate; check before blaming a run
papersynth cost runs/demo --by-providerscripts/run.sh wraps the whole sequence; scripts/run_llm.sh does the same against a
local Ollama or vLLM endpoint, which has no quota and no per-minute ceiling.
docker compose -f docker/docker-compose.yml run --rm papersynth doctor
docker compose -f docker/docker-compose.yml run --rm papersynth \
run --papers 1810.04805 --objective "..." --out runs/demoBrings up GROBID alongside, so the PDF path works without installing Java or poppler on the host.
Tests never hit a live model. LLM interactions are recorded as cassettes and replayed, so CI is deterministic and free. A nightly job re-records against the real model and diffs the resulting specs — a diff signals prompt or model drift.
docs/design-amendments.md records where the
implementation diverges from the v0.1 design, and why. Each entry is a
correction forced by something a run actually did — including one case where a
measurement in the design turns out not to be possible, and the design's claim
was removed rather than the number faked.
papersynth/schemas/*.json are the public contract, versioned semantically and readable without touching Python. spec.schema.json in particular is treated as this system's API: downstream agents pin against it, so it is additive-only within a minor version.
MIT. See LICENSE.