Repository navigation
Fetch every PROTEUS dataset through fwl-io with mirror fallback - #960
Conversation
…t to DataverseNL The monochromatic aerosol scattering tables (Zenodo 19294180) are now a PROTEUS manifest dataset, atmos_clim.scattering, with its DataverseNL mirror 10.34894/6Z8Y0Q; `proteus get scattering` fetches it through fwl-io and the AGNI wrapper lists the .mon species from its version directory. The exoplanet catalogue, the Zeng mass-radius relations, the Hammond surface albedos and the Seager EOS carry their DataverseNL pins as well, so fwl-io falls back to the mirror when Zenodo fails. With that, nothing in PROTEUS uses the whole-record Zenodo downloader or its OSF fallback: download(), the zenodo_get and OSF helpers, the source map and the Spada OSF fallback in download_stellar_tracks are gone, together with the zenodo-get, zenodo-client and osfclient dependencies. The stellar tracks come from MORS, whose fwl-mors 26.10.6 manifest pins their mirrors. The floors move to fwl-io 26.10.6 and fwl-mors 26.10.6. The reference data, troubleshooting and contributing pages describe the single fwl-io path, and the solar download error points to `fwl-io check proteus` instead of logs that are no longer written.
…istry The retry constants of the removed downloader, the empty test sections and the unused OSF storage helper are gone, as is tests/data/test_download_robustness.py, whose remaining tests were skipped and imported names that no longer exist. A skip decorator that had drifted onto the PETSc wrapper test is removed, so that test runs again. New tests pin the scattering registry (17 .mon files and one digest) and check that the fetcher PROTEUS builds passes the DataverseNL pin to fwl-io. The melting-curve no-op test checks that nothing is fetched, the Seager test no longer patches a function it never reaches, and download_stellar_tracks maps the track name to its MORS directory explicitly. The root contributing guide, the proteus.data docstring, the CI cache comment and the reference data page describe the DataverseNL mirror and say how to fetch the scattering tables before an offline run.
…ot keep download_stellar_tracks rejects a name other than Spada or Baraffe before MORS downloads anything. The solar download error no longer points to `fwl-io check proteus`, which covers only the datasets PROTEUS declares, and the troubleshooting page says so. The MORS integration helper looks for the Spada grid where MORS places it. The manifest header and both contributing guides describe the manifest route for new data, the reference page lists the scattering tables among the datasets relocation cannot move, and the manifest tests pin all 17 scattering file names and the full fetcher call.
The reference page says that the fwl-io copy of the scattering tables decides which .mon species PROTEUS offers, while AGNI reads the tables from its own res folder, and the upgrade steps say to delete the old scattering folder. Both contributing guides describe new data with the same manifest route. The solar download error says what to do next, the MORS integration helper skips when the MORS directory cannot be resolved, the AGNI wrapper imports dataset_dir once, and two test assertions are split or shortened.
… for aerosol runs FWL_IO_FLOOR now equals the fwl-io requirement, 26.10.6, so an upgrade message never names a version below what PROTEUS installs, and the test requires the two to be equal. The AGNI wrapper resolves the scattering dataset only when aerosols are enabled, and a failure to resolve it is the same warning as a missing directory. The MORS integration helper skips only when the track directory cannot be resolved on disk, so a stale MORS fails the test. The reference page lists three upgrade steps, says the old Zenodo logs and sidecar files can be deleted, and notes that PHOENIX has no mirror yet; the contributing guides name fwl-io mirror-publish.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #960 +/- ##
==========================================
- Coverage 93.57% 93.56% -0.01%
==========================================
Files 127 127
Lines 21186 20738 -448
Branches 3597 3479 -118
==========================================
- Hits 19824 19404 -420
+ Misses 941 921 -20
+ Partials 421 413 -8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Eleven unit tests build Proteus, which refuses to start without FWL_DATA. They passed only because two test modules set FWL_DATA when imported, and that value leaked into the whole session; with those modules gone, the pixi job, which sets no FWL_DATA, failed them. An autouse fixture now sets FWL_DATA to an empty session directory when the environment has none, so these tests no longer depend on which modules are collected.
The default data root is now a fresh directory per test, so a test that writes into FWL_DATA cannot leave files for a later one.
|
This is awesome - thanks @timlichtenberg. A pull request which has a net reduction in the amount of code in PROTEUS is something I aspire to! Please could you also add this Zenodo repo to the manifest? It contains aerosol refractive index data and should be pulled whenever aerosols are enabled in the AGNI config section. Thank you! https://doi.org/10.5281/zenodo.23000221 |
Hear, hear! Is this an effect of the new AGENTS.md you think? We should try giving an agent with the old md the same prompt (could maybe automate this for a few PRs actually). |
The key is now atmos_clim.scattering.socrates_aerosols, so the data lands in atmos_clim/scattering/socrates_aerosols/r19294180 and every atmos_clim dataset sits one level below its category. The name says what the 17 tables are: monochromatic (.mon) aerosol scattering data in the SOCRATES format, most of them from the SOCRATES source.
Perhaps a bit, but not alone for sure. This is more of a result of the centralisation into fwl-io, so we code moved into the fwl-io repo instead of being replicated here. So in total it's only a net reduction in so far it does not duplicate the same downloads for every repo. |
|
@nichollsh done in #974: PROTEUS now fetches the refractive indices (Zenodo 23000222, the current version of 23000221) whenever aerosols are on and points AGNI at them through |
Brings in the prescribed orbital migration (#937), the CALLIOPE cold-start seed (#958), the fwl-io dataset fetch and stellar spectra changes (#960, #964, #967, #968), the PETSc download (#965), Zalmoxis Anderson acceleration with fwl-aragog 26.10.08 (#971) and the CI changes (#955, #970). The schema invariant tests keep both the trapping and the migration validators, and the golden trajectory is main's re-recorded one with the trapping columns added.
Description
PROTEUS now fetches all of its own input data through fwl-io, with each Zenodo record pinned and its DataverseNL mirror as the fallback, and the old
zenodo_get/OSF downloader is gone. Part of FormingWorlds/fwl-io#58 (FormingWorlds/fwl-io#58).atmos_clim.scattering.socrates_aerosols(the .mon aerosol tables in the SOCRATES format, under$FWL_DATA/atmos_clim/scattering/socrates_aerosols), pinned to its mirror10.34894/6Z8Y0Q. PROTEUS reads only the.monspecies names from it; AGNI reads the tables from its ownres/scattering. The names are looked up only when aerosols are enabled, and a missing dataset gives the same warning as before.utils/data.pyloses the legacy downloader:download, the Zenodo and OSF helpers, the source markers,DATA_SOURCE_MAPand the Spada OSF fallback. The stellar tracks come through MORS, which has its own mirror fallback.zenodo-get,zenodo-clientandosfclientare no longer dependencies.fwl-io>=26.10.6andfwl-mors>=26.10.6, and the upgrade messages name the same version.$FWL_DATA/scatteringfolder can be deleted.The tests of the removed code go with it; every removed test either covers a removed function or checked the removed legacy map. Two of the removed test modules set
FWL_DATAwhen they were imported, which leaked into the whole session: on main, tests that buildProteuspass only when one of those modules is collected first, and fail on their own. A fixture intests/conftest.pynow gives each test an empty data root of its own when the environment sets none.Validation of changes
.monspecies are found in the new location.Tested on macOS with Python 3.12.
Checklist