feat: reconcile config with runtime distributed state

[Jayce-Ping]

Summary

• Extract shared env var mappings into utils/env_utils.py as a single source of truth (previously duplicated in cli.py and a now-removed logger/runtime_config.py)
• Add reconcile_config() that patches the Arguments dataclass in-place with actual runtime values from Accelerator + cluster env vars
• Call it once in load_trainer() after Accelerator creation — before any consumer (model adapter, trainer, logger) reads config — so config.to_dict() everywhere reflects the true distributed state (num_processes, process_index, num_machines, gpus_per_node, etc.)
• Logger module remains completely decoupled from accelerate

Test plan

• Verify from flow_factory.utils.env_utils import reconcile_config imports cleanly
• Verify from flow_factory.logger import load_logger works without accelerate at import time
• Verify Arguments().to_dict() excludes runtime fields when they are None
• Run single-node training and confirm W&B/SwanLab/TensorBoard config shows correct num_processes
• Run multi-node training and confirm env-derived fields (num_machines, gpus_per_node, machine_rank, main_process_ip) appear in logged config

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR centralizes multi-node environment variable handling and ensures the runtime Arguments config reflects the actual distributed state after Accelerator initialization, so downstream consumers (especially logging) observe consistent, correct values.

Changes:

• Added utils/env_utils.py with shared env-var mapping/lookup and a reconcile_config() helper to patch runtime distributed fields into Arguments.
• Updated load_trainer() to call reconcile_config() immediately after creating the Accelerator.
• Updated CLI and multinode docs to rely on the shared env-var mapping utilities and document runtime reconciliation behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/flow_factory/utils/env_utils.py	Introduces shared env var mapping + reconcile_config() to sync config with runtime distributed metadata.
src/flow_factory/trainers/loader.py	Calls reconcile_config() after Accelerator creation so subsequent components read reconciled values.
src/flow_factory/hparams/args.py	Adds runtime distributed fields (process indices, machine metadata, master IP) to Arguments.
src/flow_factory/cli.py	Switches CLI env detection to use shared ENV_VAR_MAPPINGS / env_lookup.
multinode_examples/README.md	Documents that configs are reconciled at runtime for accurate logged metadata.

Comments suppressed due to low confidence (3)

src/flow_factory/utils/env_utils.py:93

• When num_machines is set, gpus_per_node is computed via integer division (num_processes // num_machines) without validating that num_processes is divisible by num_machines. If the values ever disagree, this will silently truncate and log incorrect metadata; consider checking for a remainder and either raising/logging, or preferring an explicit gpus_per_node env var when present.

            num_machines = int(num_machines_str)
            config.num_machines = num_machines
            if num_machines > 0:
                config.gpus_per_node = accelerator.num_processes // num_machines
        except ValueError:

src/flow_factory/utils/env_utils.py:80

• Invalid integer values in cluster env vars are silently ignored (except ValueError: pass). This makes misconfiguration hard to debug and can leave defaults in place without any indication; consider logging a warning (including the env var name/value) when parsing fails.

    port_str = env_lookup("master_port")
    if port_str is not None:
        try:
            config.main_process_port = int(port_str)
        except ValueError:
            pass

src/flow_factory/cli.py:48

• num_machines_str comes from env vars and is cast with int(...) without any error handling. If a scheduler sets a non-integer value, the CLI will crash before printing a helpful message; consider wrapping this parse (and the other int(...) parses in this function) with a try/except that logs a clear error and falls back to {}.

    master_ip = env_lookup("master_ip")
    num_machines_str = env_lookup("num_machines")

    # Consider it a multi-node environment only when both master_ip and num_machines > 1 are present
    if not master_ip or not num_machines_str:
        return {}

    num_machines = int(num_machines_str)
    if num_machines <= 1:
        return {}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.