feat: PyPSA-EUR pipeline consolidation, fr225_400 dataset, full test coverage

[marota]

Summary

Consolidates the PyPSA-EUR OSM→XIIDM conversion pipeline into scripts/pypsa_eur/, ships the canonical pypsa_eur_fr225_400 dataset, and adds end-to-end test coverage so the pipeline can be iterated on without regressing silently.

Three logical changes, in three commits plus a merge from main:

1. Layout regeneration fix (82b34f1) — grid_layout.json was produced with inconsistent projection / VL key assumptions. Regenerated with correct Mercator math so layouts line up with the XIIDM VL IDs.
2. Pipeline modularization (4680689) — convert_pypsa_to_xiidm.py is rewritten as a 5-step pipeline with a shared PipelineContext (load → network → calibrate → n1verify → metadata). fetch_osm_names.py is parameterized with --voltages, --output-dir, --cache-from. New regenerate_grid_layout.py + 37-test suite covers Mercator math, key matching, and cross-network consistency.
3. Consolidation + coverage + dataset (0baa3e2) — every PyPSA-EUR script now lives under scripts/pypsa_eur/, a top-level build_pipeline.py orchestrator chains the five steps, and thermal calibration + N-1 overload reporting are extracted into their own tools. Hardcoded /home/marotant/... paths are gone.

Pipeline layout (post-merge)

scripts/pypsa_eur/
├── build_pipeline.py              # orchestrator — chains 5 steps, CLI flags propagate
├── fetch_osm_names.py             # step 1 — OSM Overpass query for substation names
├── convert_pypsa_to_xiidm.py      # step 2 — PyPSA network → XIIDM
├── calibrate_thermal_limits.py    # step 3 — N-1 peak-driven thermal-limit calibration
├── add_detailed_topology.py       # step 4 — double-busbar topology + coupler actions
├── generate_n1_overloads.py       # step 5 — N-1 security-analysis report
├── regenerate_grid_layout.py      # standalone layout regen (Mercator)
├── build_pypsa_eur_network.py     # upstream PyPSA-EUR invocation
├── verify_detailed_topology.py    # topology sanity check
├── add_limits_and_overloads.py    # legacy combined tool (kept)
├── conftest.py                    # shared pytest fixtures (network-agnostic)
└── test_*.py                      # see test matrix below

Top-level entrypoint:

python scripts/pypsa_eur/build_pipeline.py \
    --voltages 225,400 \
    --n1-peak-pct 130 \
    --min-branches 3
# → writes to data/pypsa_eur_fr225_400/ by default

Test coverage

All tests live under scripts/pypsa_eur/ and share a single conftest.py so no file re-defines the network / contingencies / layout fixtures. Counts are derived from artifacts on disk via an expected_counts fixture — nothing is hard-coded to 398 or 192 anymore, so the suite works on both pypsa_eur_fr400 and pypsa_eur_fr225_400 via --pypsa-network <name>.

File	Tests	What it covers
test_pipeline.py	end-to-end	XIIDM integrity, bus / generator / load counts, VL mapping, dispatch distribution
test_n1_calibration.py	calibration	130 % peak invariant, convergence rate, line-count parity
test_grid_layout.py	layout	VL coverage, coordinate bounds, key-format match to XIIDM
test_regenerate_grid_layout.py	Mercator	37 tests — projection math, key matching, cross-network consistency
test_build_pipeline.py	orchestrator	step selection (--steps, --from-step, --skip-osm), arg propagation, failure handling, --help
test_calibrate_thermal_limits.py	XIIDM rewriter	_patch_xiidm_limits regex, XML parseability, _overload_fraction counting
test_generate_n1_overloads.py	N-1 reporter	_get_line_limits_from_xml, module constants, CLI surface

207 passed on the non-slow subset against pypsa_eur_fr225_400.

pypsa_eur_fr225_400 dataset

Force-added (the data/ rule in .gitignore would otherwise drop it) so the canonical network ships with the repo:

• 1196 buses · 1861 lines · 361 couplers · 2351 actions
• 130 % N-1 peak loading (calibrated)
• Includes actions.json, bus_id_mapping.json, grid_layout.json, line_id_names.json, n1_overload_contingencies.json, network.xiidm, osm_names.json, vl_next_node.json

Merge note

This branch is behind main by 138 non-pypsa commits; merged origin/main with -X ours so our pypsa work wins on the five overlap points (data/pypsa_eur_fr400/actions.json, data/pypsa_eur_fr400/network.xiidm, docs/data/pypsa-eur-osm-to-xiidm.md, scripts/pypsa_eur/build_pypsa_eur_network.py), and took our side on two modify/delete conflicts (scripts/convert_pypsa_to_xiidm.py, scripts/test_grid_layout.py — both moved to scripts/pypsa_eur/). Non-pypsa main changes (backend refactor, frontend tests, docs reorganization, parity scripts, code-quality gate, …) flow through unchanged.

Test plan

• pytest scripts/pypsa_eur/ -m "not slow" — expect 207 passed
• pytest scripts/pypsa_eur/ -m slow — expect all passing (pypowsybl-heavy, ~several minutes)
• python scripts/pypsa_eur/build_pipeline.py --help — expect usage with --voltages, --steps, --from-step, --skip-osm, --n1-peak-pct, --min-branches
• python scripts/pypsa_eur/build_pipeline.py --voltages 400 --steps 5 on an existing data/pypsa_eur_fr400/ — expect fresh n1_overload_contingencies.json
• Backend smoke: load data/pypsa_eur_fr225_400/network.xiidm via the React UI, run a single contingency, confirm overloads render
• CI green on the merged tree (no regression in the 138 upstream commits)

🤖 Generated with Claude Code