Independent GitHub Actions → SSH → Slurm → Apptainer builds on SAI.
The daily workflow independently tracks ABACUS, CP2K, DeePMD-kit, LAMMPS, GPUMD and Quantum ESPRESSO,
including their development branches and latest release/prerelease versions.
Development SIFs pass build, scientific and real module-launch checks on every supported
partition before their public *-beta-later module rotates. Failed or partial matrices
retain the previous accepted version. Native /opt installation remains administrator-driven.
See current delivery status and using the beta modules.
The QE profile covers its four-partition NVHPC/MPI builds,
V100 CUDA/OpenACC support, and SCF/phonon/density acceptance.
NEPAdapters 1.0.5 has a separate manual SIF build/acceptance workflow for adding NEP CPU and CUDA Kokkos pair styles to the installed LAMMPS 30Sep2026 release. It reuses that LAMMPS build and provides a minimal, administrator-installed addon module.
The application controller does not mount host source/build/install trees writable. A build uses the pre-provisioned ~1.9 MB base SIF plus one sparse ext3 image. Git checkout, CMake, compiler temporary files, installation and final filesystem assembly all happen inside that image.
The successful artifact is one read-only SquashFS-backed SIF. It contains
/opt/apps/<software>/<stem>-<track>-<build_id>/<PARTITION> at exactly that path. The installed
administrator dependencies are reused read-only at their original paths:
/opt/devtools, /opt/modules, /usr, /lib, /lib64.
Neither /opt nor /opt/apps is bound over the installation. Site application dependencies
are mounted read-only at their exact existing version directories. See copy-only deployment.
The trusted entrypoint and bare repository are mounted read-only. Host home, working directory, site-wide bind paths and host temporary directories are disabled; the build also has a private PID namespace and a network namespace with no external network. This is filesystem/process containment using the shared host kernel, not a VM or a guarantee against kernel exploits.
Standalone LAMMPS also binds a private compiler cache writable, separately for each partition. It reuses individual objects only after matching source/header contents, compiler options and toolchain; each build still starts fresh and runs normal acceptance. The build also applies the tested upstream host-compiler optimization for C++ sources that do not use Kokkos. See the compiler-cache policy and qualification.
Every partition is a separate native build target. Configuration and compilation run on
the selected compute node with GCC -march=native -mtune=native or NVHPC -tp=host.
Actual compiler flags, hardware and module selection are recorded.
| Partition | Target | Native CPU profile | Site MPI/BLAS ISA | GPU |
|---|---|---|---|---|
| DSPRHBM | dsprhbm |
Sapphire Rapids | AVX-512 | none |
| 4V100 | 4v100-avx512 |
Zen 4 | AVX-512 | sm70 |
| 16V100 | 16v100-avx2 |
Zen 3 | AVX2 | sm70 |
| 8V100V0 | 8v100v0-avx512 |
Skylake AVX-512 (Gold 6146) | AVX2 | sm70 |
The last row is intentional: Gold 6146 has AVX-512 but lacks VNNI. The site's auto modules select compatible AVX2 dependencies; an AVX-512 CPU flag is not permission to load the newer Zen 4 dependency build. Every target verifies its actual selection.
Successful builds remove their ext3 work image after verifying the final SIF. Failed builds retain just that single image for diagnosis, not an expanded sandbox. The small base SIF is provisioned separately; the workflow fails clearly if it is absent and does not silently create an on-host sandbox.
Current builds and evidence live below /org/org-open-sai/proj-sai-hpc-software,
using Slurm account proj-sai-hpc-software. Historical home data remains in place.
Public beta modules, immutable SIF copies and their launchers are maintained under
/home/stardust/sai-hpc-software for use through the static /opt gateways.
Builds use project storage, including source caches, overlays, compiler
temporary files and Apptainer temporary/cache directories. Container
/workspace/tmp is backed by the per-run work.ext3 file in the project; host /tmp
and /var/tmp are not mounted into builds. Shared-filesystem free-space reports
are not treated as a user storage limit: submissions and running builds do not
reserve estimated disk growth or wait for a free-space threshold. Actual write
failures remain build failures with their logs and checkpoints retained.
containers/base/minimal-v1.sif
containers/software/<software>/<track>/<build_id>/<PARTITION>/<run-id>.sif
cache/repositories/abacus/ # bare Git objects, never checked out on host
controller/<controller-sha>/<run-id>/ # trusted code snapshot, never overwritten by another run
runs/<run-id>/input/ # verified compressed bundle parts when needed
runs/<run-id>/results/ # Slurm log, state, artifact checksum
runs/<run-id>/runtime/ # Apptainer runtime work, not source/build/install
runs/<run-id>/work.ext3 # one temporary file, retained on failure
runs/<run-id>/artifact.path # candidate SIF location after build verification
modulefiles/apps/abacus/<track>/<build_id> # partition selector for accepted SIFs
runtime-tests/<run-id>/ # bounded multi-node acceptance inputs, logs and rank evidence
Old sandbox-based runs are legacy leftovers; this controller does not delete those automatically. Do not infer success from earlier smoke images or a GitHub validation-only run.
Dispatch Build HPC software with a comma-separated subset of development,prerelease,release
and registered targets. Each channel resolves live to its actual upstream commit.
No synthetic/orphan commits are substituted. Development install and module names use
<branch>-<commit-UTC-date>-g<source-sha12>-r<recipe-sha12>, for example
develop-2026-09-13-g0123456789ab-rabcdef012345. A retry on another day retains the same
build identity. SIF filenames also include the submission UTC date before the source SHA;
the Actions run ID and attempt distinguish retries.
The daily tracker runs at 02:23 UTC (10:23 Asia/Shanghai).
profiles/software-tracking.json documents the shared software/channel/partition contract.
Development rebuilds each day, even when the source commit has not changed. Release and
prerelease versions are attempted once per upstream version, including unsuccessful attempts;
select retry_releases to retry them explicitly. A skipped release is not relabeled as accepted.
Artifact acceptance requires matching source, recipe, SIF checksum and original raw scientific
evidence; legacy verified: true alone is insufficient. Each software promotes its complete
accepted development matrix independently, so a failed release track does not block that beta.
A100 is not registered for ABACUS publication acceptance and is rejected before scheduling.
Build verification creates a candidate, with build_verified: true but
verified: false, published: false. It does not update current.sif or the module.
Only software_controller.py publish RUN_ID, after all required acceptance has passed,
exposes the image. A failed acceptance leaves the previous published image unchanged.
CP2K publication requires all seven candidate scientific cases on one and two nodes,
followed by native export readback. An untested candidate cannot become an accepted cache hit.
The gate binds the build, image, launcher, verifier, job, raw rank traces and scientific
outputs by checksums; an exit code of zero without those results does not pass.
Cache hits upload zero source bytes. Cache misses bundle only changes against an available ancestor (or a full seed if no ancestor exists). The bundle is gzip-compressed, split into exactly eight byte chunks, and transferred concurrently. Each part and the full compressed and decompressed streams have checked sizes and SHA-256 hashes. The host receiver verifies Git prerequisites before importing into the bare cache under a lock; it never runs source code. All child transfer failures are checked.
An administrator can seed the cache from an existing full-history local Git checkout:
python3 controller/source_cache.py pack /path/to/local/repository upstream-sha ./seed
# Upload manifest.json and source.part.00 through source.part.07 to an input directory.
# On SAI, run the trusted controller already provisioned outside any build container:
python3 controller/source_cache.py receive \
/org/org-open-sai/proj-sai-hpc-software/cache/repositories/abacus \
/org/org-open-sai/proj-sai-hpc-software/runs/manual-seed/inputUse --base <cached-ancestor-sha> when making an incremental seed.
No source checkout or build/install directory is created by the receiver.
The public host key is versioned in .ci/slurm/known_hosts.
REMOTE_SSH_PRIVATE_KEY and REMOTE_USER are Actions secrets
(current deployment: the repository's hpc Environment).
Reusable workflows resolve these credentials inside their hpc jobs; caller
secrets are optional. Missing credentials stop the SSH preparation step before
any remote action. Result jobs require no environment credentials.
No private key is committed. Only scheduled and manually dispatched trusted runs access the SSH key;
push and PR runs only validate. Keep the hpc Environment limited to trusted branches.
Targets: dsprhbm (DSPRHBM), 4v100-avx512 (4V100),
16v100-avx2 (16V100), 8v100v0-avx512 (8V100V0). The A100 recipe remains in the code, but ABACUS publication is disabled
until an A100 runtime acceptance is registered.
Pass a comma-separated subset to dispatch. CPU (DSPRHBM) is the default acceptance target.
Every build runs independently with its own overlay, logs and SIF path. GitHub retains logs
and the SAI artifact location, while the container itself stays on SAI.
The generated module is loaded inside the Slurm allocation, so the site *-auto
dependency modules inspect the actual compute-node CPU. The trusted abacus command then
selects the SIF from SLURM_JOB_PARTITION. The host Open MPI launches one wrapper per rank;
each wrapper enters the same read-only SIF and runs its ABACUS binary. GPU job example:
The version module selects a partition-local, immutable module fragment that pins
both the accepted SIF and its tested launcher. Publishing another partition does
not replace this pair or its dependencies. A loaded module retains that pair until
unloaded/reloaded; loading outside a Slurm allocation is rejected. module show
remains available without an allocation. Legacy artifacts need publication with
this contract before their partition has a current.module entry.
source /etc/profile.d/lmod.sh
module use /home/stardust/sai-hpc-software/modulefiles/apps
module load abacus/<track>/<build_id>
source /opt/sai_config/mps_mapping.d/${SLURM_JOB_PARTITION}.bash
export MAP_OPT SLURM_EXPORT_ENV=ALL
export OMPI_MCA_plm_slurm_args=--external-launcher
export PRTE_MCA_plm_slurm_args=--external-launcher
export TMPDIR="$SAI_SOFTWARE_ROOT/runtime/jobs/$SLURM_JOB_ID/mpi"
mkdir -p "$TMPDIR"
mpirun -np "$SLURM_NTASKS" --map-by "$MAP_OPT" abacusRuntime does not use the build container's network isolation or fakeroot. --nv exposes
the Slurm-assigned NVIDIA devices and matching host driver libraries; MPI/network setup
remains host managed. The launcher binds only the calculation directory writable, keeps
the SIF and /opt/devtools read-only, and puts Apptainer runtime files below
/home/stardust/sai-hpc-software/runtime/jobs. It rejects an unknown partition rather than
falling back to an incompatible image. The host MPI session directory must also stay below
the project runtime roots and is bound at the same absolute path so PMIx shared-memory
metadata remains visible to ranks inside the container.
Every new DSPRHBM build is followed by a two-node CPU PW SCF (8 ranks/node, 2 threads/rank).
Every new precise V100 build is followed by a two-node, one-rank-per-GPU scientific smoke.
This deliberately avoids the site's multi-rank-per-GPU MPS path, which currently uses
host /tmp. The acceptance records rank/hostname/image selection, requires two distinct
nodes, exercises CUDA and MPI in a short PW SCF case, and requires SCF convergence and
an energy error of at most 1e-5 eV.
GPU publication also requires two real ABACUS feature cases, packaged inside each new SIF:
- Si2 LCAO with
ks_solver cusolvermp: actualcusolverMpSygvd/Hegvdlogging, convergence and reference energy. Buffer-size queries and--infodo not count as execution. - GaAs PW BPCG with
kpar 1, bndpar 2: actual NCCL collective logging withnranks=2, two distinct nodes, convergence and reference energy. NCCL initialization alone does not count. This decomposition exercises AllGather; AllReduce/Broadcast also count when actually logged.
These are submitted by gpu_feature_controller.py; both runtime controllers use pinned
candidates directly and therefore do not need a module to be published before testing.
Use the published path from runs/<run-id>/artifact.path, not a legacy image:
ssh SAI-stardust
source /etc/profile.d/lmod.sh
module load apptainer/1.4.4
apptainer exec --cleanenv --containall --no-home \
--no-mount bind-paths,home,cwd,tmp,hostfs --pwd / \
--bind /usr:/usr:ro --bind /lib:/lib:ro --bind /lib64:/lib64:ro \
--bind /opt/devtools:/opt/devtools:ro \
/home/stardust/sai-hpc-software/containers/software/abacus/TRACK/BUILD_ID/PARTITION/RUN.sif \
/usr/bin/find /opt/software -maxdepth 7 -type fThe shared exporter packages each install as one complete folder, including
share/sai/manifest.json and modulefiles/. It does not create /opt/sai-delivery.
An arbitrary export destination is staging, not a claim that compiled-in paths are relocatable.
See native delivery for usage and tested limits and
workflow status for per-software evidence. CP2K 2026.2 has passed
physical native science; timings are descriptive, with no retired 2026.1 baseline required.
The SIF records the upstream SHA, module list and CMake cache below the installation's
share/sai/ directory. Loading its recorded modules is required to run software
that dynamically links to the cluster environment. Administrative deployment into the host
/opt is performed separately by an administrator using the inventoried deployment package.
The accepted four-partition CP2K 2026.2 release using repaired shared ELPA was
installed with the administrator upgrade package.
It preserves exact build prefixes, and cp2k/2026.2-auto selects the new installation
inside Slurm allocations. File readback and all 56 physical native calculations passed;
see the independently revalidated evidence.
For a read-only runtime, source share/sai/runtime-env.sh inside the container
instead of initializing Lmod there; it records the original absolute dependency
paths without needing writable temporary files. Packaging/verification failures
can be retried using the workflow's resume_run input: this moves the old run's
single ext3 image into the new run and repacks the existing installation. The
source SHA, version, target and recipe fingerprint must match, and active jobs cannot be resumed.
Start with the ELPA reproduction and integration guide for the partition map, pinned source, build commands, current acceptance and CP2K usage.
ELPA 2026.02.001 deployment provides separate
AVX2 and AVX512 builds of the site's GNU/CUDA configuration. This standalone dependency
recipe runs inside Apptainer with task-owned home directories bound for build storage and
the exact future /opt/devtools/elpa/.../nvidia prefix. Site dependencies are read-only.
Final SIFs are checked with CPU/GPU, real/complex, MPI/OpenMP numerical solves before
preparing an inventoried administrator handoff and an automatic ISA module selector.
The installed physical libraries passed 144 CPU/GPU solves across the three V100
partitions. The NVIDIA library's explicit C pointer-detection API requires a CUDA
driver on DSPRHBM. A CPU-only supplement
now passes all 24 explicit-API CPU solves without that dependency; it upgrades the
existing elpa/2026.02.001-2603-gnu-auto module to select CPU on DSPRHBM and the
accepted GPU libraries on V100. Administrator installation is complete. Physical
jobs 1861947–1861950 passed all 168 CPU/GPU solves across the four partitions,
including the original explicit C APIs on DSPRHBM, fresh Fortran consumers and
actual ELPA/MPI/BLAS/HCOLL loader evidence. See the
native acceptance report.
The subsequent GPU lifecycle audit found
a missing upstream MPI communicator free. The small numerical results stand;
repeated long-running GPU diagonalization needs the archived repair and renewed qualification.
The repair replacement package
now contains three rebuilt variants with upstream PR #85. All four partition
checks passed, including 32 communicator creations and 32 frees per GPU rank.
The administrator preflight passed; physical replacement and subsequent native
acceptance are still pending. Repair evidence.
python3 -m unittest discover -s tests -v
for script in controller/*.sh controller/*.sbatch; do bash -n "$script"; doneTests cover real full/incremental Git cache reception, missing prerequisites, transport corruption, path injection, generated Slurm syntax and the read-only bind policy.