Skip to content
quangng2000Public

About

RMFSR speech restoration demo: PyTorch model, notebook, data pipeline, and GPU training configuration.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

RMFSR

Listening demo · Page source and reproduction notes

Current training pseudocode and JVP explanation — 18 October snapshot with frozen wavJEPA and the full MeanFlow loss; waveform-fidelity auxiliaries disabled.

This repository contains an independent implementation attempt and opt-in experimental architectures. It is not the authors' released model and its results have not been validated against the paper. No official weights or inference code were found in the RMFSR demo repository. The source paper is Braun, 2026, arXiv:2605.16251.

Current status, October 4, 2026

  • Implemented the five-level causal U-Net, four-layer TCN, frequency attention, SnakeBeta, data-prediction MeanFlow loss, and separate streaming state for each flow step.
  • Ran 100 training steps on the Mac MPS GPU, using one EARS training speaker and a different validation speaker. A third speaker is reserved for pilot testing. This is a plumbing/numerical pilot, not full training.
  • The September 30 baseline passed 66 regression tests, covering training math, target EQ, verified datasets, actual restoration validation, causality, codecs, accumulation and checkpoint resume. A historical full-width (legacy-v1), effective-batch-16 MPS numerical smoke completed one update plus 1/2/5-step validation and review audio; this establishes execution, not restoration quality.
  • Evaluated the checkpoint at 1/2/5 flow steps against the existing Opus/tPLCnet comparison inputs. The pilot does not yet fill speech gaps; its output is essentially the damaged input. Do not interpret this as evidence against the published method.
  • New runs use decoder outputs [256,256,128,64,64] (mirror-v2). The 100-step pilot used the older shifted decoder; it remains a historical result and cannot resume into the new architecture.
  • Added an optional capacity/efficiency study, with measured counts and a fair comparison plan in ARCHITECTURE.md.
  • Added efficient-v1, an experimental engineering variant aimed at the paper's compute budget. Its recommended configuration has 7,742,598 parameters and counts 0.911 GMAC/audio-second/NFE for four-second calls, or 1.192 for one-frame calls. These operation counts do not establish runtime or audio quality; the author architecture has not been recovered.
  • All 76 tests pass after integration, including ten new checks for model selection, checkpoint round trips, legacy compatibility, compact attention accounting, causality, streaming, JVPs and a tiny save/resume/evaluation smoke test.
  • The proposed full run has not started. Full dataset preparation is pending, and architecture parity remains unresolved.

Paper-to-code audit

Area Implementation Status
Backbone width Encoder [64,64,128,256,256]; decoder outputs [256,256,128,64,64]; depthwise expansion 2 Literal stage-output mirror; exact author tensors unavailable
Kernels Encoder 3×3, decoder 3×2, TCN 1×11 Matches stated kernels
Flow path Degraded mean; sigma .3 → 1e-8; compressed complex spectrum, exponent .3 Matches stated formulation
Training Model-derived instantaneous JVP tangent; stop-gradient JVP; data-prediction loss Independent equation implementation, analytically tested
Time sampling Logit-normal mean .4, SD 1; active figure1-cosine-approx reproduces both plotted Figure 1 curves Prose calls the diagonal curve sigmoid-based; formula unpublished. Epoch-to-update mapping remains an assumption
Architecture size 6,112,710 parameters, measured 7.156 GMAC/audio-second/NFE Paper reports 7.8M / 1.22 GMAC/s; substantial mismatch, not validated parity
Context 218 frames; 2.17 s past context at 10 ms hop Paper reports 2.13 s; exact dilations unspecified
Spectrum 16 kHz, 320-point FFT, sqrt-Hann, 20 ms window, 10 ms hop Sample rate/hop/window details chosen for existing phone benchmark
Normalization Divide FFT by window sum, inverse on synthesis Explicit choice: bounds spectral magnitude for bounded audio; paper's “power normalization” divisor is unspecified
Attention/norm Four-head, frame-local frequency attention; channel-only normalization; optional TCN attention study Exact attention/normalization not specified by paper
Noise Complex spectral Gaussian with 1/f energy, frequency-average unit variance DC floor and absolute normalization are explicit choices
Optimizer/training length AdamW, LR 1e-4, proposed 300k steps, EMA .999 Our starting configuration; not reported paper hyperparameters
Initialization Zero-initialized output residual around degraded input Our stabilization choice; not reported by paper
Augmentation Every listed family implemented; finite image-source RIR, local codec round trips, LTAS target EQ Probabilities, reflection order, filter/compressor parameters and EQ limit are approximations
Pilot data EARS p001 train, p002 validation, p003 test; synthetic colored background; no DAPS EQ Explicit pilot deviations; cannot establish general restoration quality
Full data EARS p001–p099 train, p100–p103 validation, p104–p107 test; DNS noise; DAPS produced LTAS Proposed speaker-disjoint split; full assets not prepared
Paper evaluation SIG2024, listening tests, MOS and WER Not reproduced; current comparison is one held-out synthetic utterance

The table describes the original mirrored backbone. The experimental efficient-v1 family is a separate architecture study, selected explicitly below. Its active grouped layers, lower-rank attention and folded bottleneck change the computation; matching a parameter or MAC budget does not establish author parity. Do not quote the paper's compute or latency results as measurements of this implementation.

Experimental efficient architecture

configs/efficient.json selects model_type="efficient-v1", groups=16, attention_rank=16 and folded_width=448. configs/efficient-small.json uses eight groups and folded width 256 for a smaller 4,532,742-parameter variant. Both retain the proposed full-data recipe from runpod-estimate.json: Figure 2 augmentation, the approximate Figure 1 flow schedule, four-second crops, effective batch 16 and 300,000 proposed updates. They choose the available device automatically. Existing configurations retain their original architecture.

# Forward-only operation counts; no dataset preparation or training.
mkdir -p runs
python -m rmfsr.profile --config configs/efficient.json --inventory-only --device cpu --seconds 4 --output runs/efficient-inventory-4s.json
python -m rmfsr.profile --config configs/efficient.json --inventory-only --device cpu --seconds 0.01 --output runs/efficient-inventory-10ms.json
python -m rmfsr.profile --config configs/efficient-small.json --inventory-only --device cpu --seconds 4 --output runs/efficient-small-inventory-4s.json

# After full EARS/DNS/DAPS preparation and device qualification; starts fresh training.
python -m rmfsr.preflight --config configs/efficient.json
python -m rmfsr.train --config configs/efficient.json --run runs/efficient

The small variant counts 0.779 GMAC/audio-second/NFE for four-second calls and 1.009 for one-frame calls. Long calls amortize conditioning computations across more frames, so quote the call length with each MAC count. One-frame counts still exclude framework overhead, normalization, nonlinear functions, memory traffic and the STFT. See ARCHITECTURE.md for the design tradeoffs and comparison protocol.

These variants require new training and different weights. The checkpoint loader reconstructs the saved model type for evaluation; resume rejects a different architecture. To evaluate a newly trained checkpoint using the existing aligned comparison bundle:

python -m rmfsr.evaluate --checkpoint runs/efficient/best-validation.pt --baseline-dir /path/to/comparison-bundle --output runs/efficient/evaluation --steps 1 2 5 --chunk-frames 1

The comparison bundle must use the existing opus_*ms directory format with aligned references, damaged inputs, masks and baseline outputs. This command evaluates a checkpoint; it does not supply pretrained weights. No restoration-quality or measured real-time claim is supported for efficient-v1, and adding these configurations does not start training or provision compute.

MeanFlow equations and choices

For clean data x0, degraded audio y, and spectral pink noise e:

xt = (1-t)*x0 + t*y + ((1-t)*sigma_min + t*sigma_max)*e
v_cond = y - x0 + (sigma_max-sigma_min)*e
u(xt,r,t) = (xt - model(xt,y,t,r)) / t
v_model = u(xt,t,t)
jvp = stopgrad(JVP(u; primals=(xt,r,t), tangent=(v_model,0,1)))
x_target = xt - t*v_cond + t*(t-r)*jvp
loss = mean((model(xt,y,t,r) - x_target)**2)

The t² weighting implicit in this x-space loss is the data-prediction interpretation of IMF. At r=t, it reduces to predicting x0 + sigma_min*e. The off-diagonal sign and model-derived tangent have an analytic regression test.

Paper Eq. 14 prints U(0,t)^gamma and then multiplies by t again. We interpret its span description using r=t*U(0,1)^gamma, making early samples concentrate near r=t. This is an explicit interpretation, not an exact transcription.

The active figure1-cosine-approx schedule was checked against vector coordinates extracted from Figure 1 in paper v1. The plotted diagonal ratio decreases from 1 to .2 by half the displayed epoch range; gamma increases from .05 to 1 by .8 of that range. Both curves match the current cosine formulas: maximum absolute discrepancy is 0.0000775 for the diagonal ratio and 0.0001020 for gamma at the extracted vertices. Section 2.4 calls the diagonal schedule "sigmoid-based" without specifying a logistic formula. A newly chosen logistic curve would therefore not establish greater fidelity to the published plot. We retain the plotted behavior and test it against independently extracted coordinates in tests/fixtures/paper-figure1-schedule.json. This does not recover the author's source code: mapping the epoch axis to fractional optimizer-update progress, and choosing a 300,000-update run, remain implementation assumptions. Historical legacy scheduling uses a different logistic approximation; it is retained only for existing configurations and is not the active full-run schedule.

An explicitly queued flow-schedule-transition.json in a run directory can accelerate the flow curriculum at the next checkpointed session resume. The request is bound to the exact configuration digest. The clock starts at the checkpoint's existing progress, then smoothly reaches Figure 1 progress .8 over 10,000 further optimizer updates: diagonal probability .2, span exponent 1. This is an experimental training choice, not the author's published update schedule. The learning-rate clock and 300,000-update target remain unchanged. The resolved clock is stored in checkpoints and flow-schedule-active.json, and the pre-change checkpoint is retained. Subsequent resumes keep the original anchor; they do not restart the ramp. No live process reload or signal is used.

No GAN, pretrained vocoder, latent autoencoder, or clean target is used at inference. Evaluation does not train on or use the missing clean samples.

Run locally

Clone and install from the repository root (Python 3.10+):

git clone https://github.com/quangng2000/rmfsr.git
cd rmfsr
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install jupyterlab nbformat ipykernel
python -m ipykernel install --user --name rmfsr --display-name "RMFSR (PyTorch)"

GSM needs libgsm (brew install libgsm on macOS; sudo apt-get install libgsm1 libsndfile1 on Debian/Ubuntu). imageio-ffmpeg supplies the MP3 encoder. CUDA hosts should use the setup in TRAINING.md.

Prepare a small, speaker-disjoint pilot and run it:

python -m unittest discover -s tests -v
python -m rmfsr.prepare --root data --split portable_pilot_train --speakers 1
python -m rmfsr.prepare --root data --split portable_pilot_validation --speakers 2
python -m rmfsr.prepare --root data --split portable_pilot_test --speakers 3
python -m rmfsr.preflight --config configs/pilot-figure2.json
python -m rmfsr.train --config configs/pilot-figure2.json --run runs/my-pilot

The pilot config selects Apple MPS. Set device to cuda or cpu in a copied config for other hosts. Data, checkpoints, generated audio and local results are excluded from Git. The notebook documents the original experiment; its result cells require the locally generated runs/pilot artifacts and external Opus/tPLCnet comparison files. A fresh clone does not include those results or pretrained weights. Open it with jupyter lab rmfsr.ipynb; use the commands above for a new pilot.

Checkpoint files include weights, EMA, optimizer, training step, data RNG, CPU/MPS/CUDA RNG, configuration, verified audio-content/partition fingerprints, and metric history. Checkpoints from before the verified-content format, corrected targets, or mirrored decoder require a fresh training run. Architecture flags are saved and checked before weights load; evaluate.py retains the original decoder when reading historical pilot weights. Only load trusted local checkpoint files. To extend training, increase steps in a copied config; preserve schedule_steps for a consistent schedule. Full mode cannot resume a pilot checkpoint as though it were full training.

Full data preparation and training

Not yet completed. Fetch the EARS dataset, DNS non-speech noise, and DAPS produced studio speech. Do not use DAPS device recordings as the studio LTAS reference. Noise input must be the curated non-speech portion, not interfering speech.

The EARS helper processes one speaker archive at a time, verifies ZIP CRC, records SHA-256 provenance, resamples to 16 kHz FLAC, and removes only its own completed temporary archive. It reuses completed speaker conversions.

python -m rmfsr.prepare --root data --split train --speakers $(seq 1 99)
python -m rmfsr.prepare --root data --split validation --speakers $(seq 100 103)
python -m rmfsr.prepare --root data --split test --speakers $(seq 104 107)
python -m rmfsr.ltas --produced-dir /path/to/DAPS/produced --output data/daps_ltas.npy
# Put/extract curated DNS noise WAVs under data/dns_noise, or set noise_dir in config.
python -m rmfsr.preflight --config configs/full.json
python -m rmfsr.train --config configs/full.json --run runs/full

Full mode requires complete documented speaker splits, DNS noise, DAPS LTAS and codec tools. It fails instead of silently using pilot substitutes. Do not reuse the pilot p003 test results as held-out evaluation if a later full run includes p003 in training; use p104–p107 instead.

Raw downloads are large: EARS release assets total ~69 GB, DNS noise archives ~39 GB, and the complete DAPS archive is 16.1 GB. Use sequential conversion and a data volume sized for the converted datasets plus download headroom.

Historical legacy-v1 measurement on this Mac: batch 4 × 4-second clips, full-width forward/JVP/backward/update, median 3.54 s/step over three timed steps. For the original batch-4 configuration without gradient accumulation, extrapolation to 300k steps is ~12.3 days, before augmentation, validation, I/O and thermal variability. This is a historical planning estimate, not timing for the corrected decoder or the paper's training requirement. Rebenchmark the chosen architecture before estimating GPU time. No paid cloud resources have been provisioned.

Streaming and evaluation

StreamingRestorer.push(samples) / .finish() accepts arbitrary waveform chunks. It carries causal STFT, overlap-add synthesis, and separate convolution history for every flow step. No normalization uses future frames. The streaming tests compare both network outputs and waveform reconstruction against the same offline computation and identical prior noise.

evaluate.py uses 10-frame chunks (100 ms) by default for faster Python benchmarking; this batches input and adds buffering delay. This is not a 20 ms end-to-end latency measurement. The frame-by-frame streaming API avoids that batching, but has not demonstrated real-time hardware performance.

Saved comparisons include original, damaged, Opus Classic/Deep PLC, tPLCnet S/L, and RMFSR NFE 1/2/5. All receive the same audio and packet-loss mask. A common playback gain is used across outputs. Metrics include whole-clip STOI/SI-SDR plus gap RMSE, gap energy and second-half gap energy. Loudness alone is not correctness; phase-sensitive error alone is not perceptual quality. The 120 ms condition exceeds the paper's stated 10–80 ms training-dropout range and is labeled a stress test.

The current eager-Python/MPS NFE=5 comparison runs slower than real time. CUDA optimization, a statistically useful test set, gap listening tests, SIG2024/MOS/WER evaluation, and mobile deployment profiling remain future work. No claim that this checkpoint beats Opus or tPLCnet is supported.

Data terms

EARS is CC BY-NC 4.0. Retain its attribution and license; these research checkpoints are not cleared for commercial deployment. DNS and DAPS have their own data terms. Downloaded audio is local and has not been uploaded or published.

Cloud training preparation

See TRAINING.md for the Figure 2 implementation audit, portable EARS/DNS/DAPS preparation, Runpod GPU comparison, benchmark scripts, and checkpoint recovery. configs/runpod-estimate.json contains the latest proposed effective-batch-16 training recipe; configs/runpod.json retains the earlier batch-4 recipe. Both use figure2-v2 augmentations. The training script now defaults to configs/efficient.json; the benchmark defaults to configs/efficient-benchmark.json (100 updates, 30-minute training-process limit). Both use effective batch 16. Synthetic timing includes gradient accumulation, and both paths select the same configured model. Validation restores fixed damaged-only inputs at 1/2/5 steps, selects best-validation.pt by five-step waveform MSE, and saves gap/intact-region metrics plus review audio. Existing pilot results remain labeled as the legacy synthetic-noise pilot.

Optional paper-1 speech-fidelity loss

configs/balanced-paper1-fidelity.json adds PESQ and negative SI-SDR supervision from Yang et al., arXiv:2512.10382, section 3.4 to the existing MeanFlow objective. Install its optional dependencies with pip install -e '.[fidelity]'. The paper explicitly links the open-source torch-pesq dependency; an official repository for the full paper implementation has not been confirmed.

The adapted objective is MeanFlow + 0.001 * PESQ + 0.0001 * negative_SI_SDR. Those weights are the paper's plain data-prediction row of Table 2, used here as starting values. The PESQ library factor is explicitly 1.0; the paper does not specify that constructor setting. Our MeanFlow parameterization, spectral normalization and loss reduction differ, so these are not calibrated weights or an exact reproduction of its EDM-preconditioned NCSN++ model.

Auxiliary losses use the instantaneous r=t clean estimate, decode it with the existing inverse compression, normalization and causal WOLA alignment, and compare its waveform to the clean training waveform. They do not treat an off-diagonal MeanFlow output as a clean endpoint or run a full inference rollout. Mixed/off-diagonal batches require one extra differentiable model call. CPU and CUDA are supported; MPS and clips shorter than 0.5 seconds are rejected.

PESQ/negative-SI-SDR, their weighted contributions, base MeanFlow loss, RMS levels and silent-example exclusions are logged separately. Undefined silent references are excluded from both auxiliaries; silent predictions are excluded from PESQ's level normalization. The base coefficient loss still supervises these examples. Neither auxiliary is a direct absolute-loudness constraint. Their sum may be negative because negative SI-SDR is negative for good estimates; use fixed held-out restoration metrics and listening, not total training loss, for quality.

Existing recipes remain disabled. Enabling this on a checkpoint requires allow_speech_fidelity_on_resume: true; the trainer preserves a pre-change checkpoint and records the transition. Subsequent resumes cannot silently alter the weights. The saved flow clock, optimizer, EMA, learning-rate clock and data recipe are preserved. This local candidate does not activate itself on a pod. No restoration-quality improvement has yet been demonstrated for this change.

Optional WavJEPA conditioning experiment

The wavjepa option adds frozen, pretrained damaged-audio frame features to the efficient model's bottleneck. A LayerNorm/768-to-64 projection and residual 1x1 convolution combine them with the CNN's frequency/time features. The fusion convolution starts at zero, preserving the source checkpoint's predictions. The adapter and restoration network train; WavJEPA stays frozen. The MeanFlow/JVP objective, optional speech-fidelity losses and time sampler remain unchanged. Features are computed once per microbatch or evaluation clip and reused for every NFE. The encoder is outside the restoration EMA and checkpoint, identified by its pinned model revision.

This is an offline experiment. The published encoder uses bidirectional attention in approximately 2-second segments. It does not preserve our causal streaming latency; streaming is explicitly rejected for this variant. Feature alignment uses the published frontend's actual convolution centers and segment length, including STFT padding/tail frames. Only the encoder copy is normalized; the damaged spectrum and clean targets retain their existing levels.

Install pip install -e '.[wavjepa,fidelity]' in a separate environment. The first encoder load downloads the authors' public checkpoint and custom code at immutable revision 6be4a5093f6c9adbbd55e00b6e5b8f067aa03345. The bundle includes unused teacher/predictor modules; these are discarded after loading.

Create a new experiment from a trusted checkpoint:

python -m rmfsr.wavjepa_fork --checkpoint runs/balanced-v1/latest.pt --run runs/wavjepa-v1

This preserves backbone/EMA weights, AdamW moments, RNG state, update count, data recipe, learning-rate clock and flow curriculum. Only the adapter starts fresh. Validation switches to offline inference and resets best-score selection; use matched offline inference for comparison with the original checkpoint. The old run is not modified. The command does not start training or touch a pod. After deciding to activate the new experiment, run:

python -m rmfsr.train --config runs/wavjepa-v1/config.json --run runs/wavjepa-v1 --resume runs/wavjepa-v1/latest.pt
python -m rmfsr.wavjepa_restore --checkpoint runs/wavjepa-v1/latest.pt --input input.wav --output restored.wav --nfe 20

Inference expects mono 16 kHz audio and writes raw floating-point WAV without playback normalization. This integration is not evidence of better restoration; judge a trained candidate against the same held-out inputs and noise seeds.

Positional time conditioning and adaptive double normalization

The opt-in conditioning_architecture: "scm-positional-adn-v1" uses the tested identity-time positional embeddings and frame-local adaptive double normalization. It replaces every conditioned convolution while preserving the full MeanFlow/JVP objective. This is an sCM-inspired adaptation to the speech CNN, not the complete sCM model or objective. Existing configurations keep their original conditioning.

rmfsr.conditioning_migration.convert_checkpoint(source, destination, request) converts a trusted width-4 production checkpoint into a separate file. The request binds the exact source hash, configuration hash, and completed update count. Conversion retains learned raw/EMA weights, data fingerprints, RNG state, and learning-rate/flow clocks. It initializes the added scale projections, resets AdamW moments once, and resets best-validation selection. Subsequent resumes retain optimizer state and verify the saved architecture and migration metadata. The first update after conversion is checkpointed before the normal interval.

The pre-step stability guard rejects nonfinite or excessive raw metrics before an optimizer update. The continuous controller does not automatically retry a failed trainer. Neither component stops or releases a GPU pod. Short stability tests do not establish long-run stability or improved restoration quality.

About

RMFSR speech restoration demo: PyTorch model, notebook, data pipeline, and GPU training configuration.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages