refactor(install): modularize install.py into engine package

[danielmeppiel]

Summary

Decompose commands/install.py from a 2905-line module (containing one 1444-line god-function) into a coherent apm_cli/install/ engine package. 56% LOC reduction in commands/install.py (2905 → 1268), zero behavioral changes, zero test regressions.

Builds on #762 (the security/UX hardening); preserves the cleanup chokepoint integration/cleanup.py.remove_stale_deployed_files invariant.

Why

install.py had become the largest and most error-prone module in the codebase. The single _install_apm_dependencies function held ~40 implicit shared locals across resolution, target detection, downloading, integration, cleanup, lockfile assembly, and result rendering, making targeted changes (like the recent #762 cleanup hardening) high-risk and hard to review.

What changed

New engine package (src/apm_cli/install/)

apm_cli/install/
  context.py             InstallContext dataclass (data backbone)
  validation.py          _validate_and_add_packages_to_apm_yml
  helpers/security_scan.py
  phases/
    resolve.py           lockfile load + auth + BFS resolution + --only filtering
    targets.py           target detection + scope + integrator init
    download.py          ThreadPoolExecutor parallel pre-fetch
    integrate.py         per-package sequential integration loop
    cleanup.py           orphan + intra-package stale (routes through #762 chokepoint)
    lockfile.py          LockfileBuilder (7 cohesive methods) + compute_deployed_hashes
    local_content.py     root primitives + local-path copy helpers
    finalize.py          stats + result construction
  presentation/dry_run.py

Thinned adapter

commands/install.py: 2905 → 1268 LOC. Now a Click adapter that re-exports engine symbols (preserving the @patch("apm_cli.commands.install.X") contract used by ~30 test sites) and orchestrates InstallContext through the phases.

Architectural invariant guard

New tests/unit/install/test_architecture_invariants.py enforces a 1000 LOC budget per module under apm_cli/install/ (with documented exceptions: 900 for phases/integrate.py, 1500 for legacy commands/install.py). Prevents future re-bloat.

How (key design decisions)

1. Engine sibling, not commands/install/ package. Turning commands/install.py into a package would name-collide and force rewriting all @patch("apm_cli.commands.install.X") sites. Keeping the Click adapter as a single .py that re-exports from apm_cli/install/ preserves all test patches verbatim.

2. Module-attribute access pattern (_install_mod.X). When a phase needs to call code that tests patch via apm_cli.commands.install.X, it imports the module (from apm_cli.commands import install as _install_mod) and references _install_mod.X so monkeypatching at the patched namespace flows through. Used for _integrate_package_primitives, _rich_success/_rich_error, _copy_local_package, _pre_deploy_security_scan, GitHubPackageDownloader. AST-verified zero bare-name calls to patched symbols in phases/integrate.py.

3. InstallContext is the data backbone. Each phase extends it with the fields it owns. Mutable dict/list fields (package_deployed_files, etc.) carry mutations across phase boundaries (cleanup mutates lists that LockfileBuilder reads). All defaults via field(default_factory=...) to prevent shared-default bugs.

4. fix(install): harden stale-file cleanup with per-file content-hash provenance (#666 follow-up) #762 invariant preserved. All cleanup paths still flow through integration/cleanup.py.remove_stale_deployed_files (3 safety gates intact). Security review explicitly confirmed; 16/16 cleanup helper tests + 32 cleanup/orphan integration tests green.

Phased execution (13 commits)

• P0 — engine package skeleton + InstallContext stub
• P1.L1-L4 — leaf extractions (validation, local_content, lockfile/_hash_deployed, security_scan)
• P2.S1-S6 — sequential extractions of resolve+targets, download, integrate (largest, 715 LOC moved), cleanup, dry-run, lockfile builder + finalize
• P3.R1 — cli-logging-ux hardening (fixed 2 BLOCKERS: Unicode box-drawing chars in security_scan.py and resolve.py that would crash Windows cp1252 terminals; 3 other ASCII fixes)
• P3.R2 — activated architectural invariant tests with realistic budgets
• P3.R3 — 3-agent parallel review (architect + cli-logging-ux + security)

Review verdict

Final 3-agent parallel review:

• Python architect: APPROVE WITH FOLLOW-UPS — clean layering, cohesive phases, faithful extraction (spot-checked _hash_deployed, _copy_local_package, integration loop), test-patch contract airtight
• CLI logging UX: APPROVE WITH FOLLOW-UPS — 5 non-blocking polish items, all 2 BLOCKERS fixed in P3.R1
• Security: APPROVE WITH FOLLOW-UPS — fix(install): harden stale-file cleanup with per-file content-hash provenance (#666 follow-up) #762 chokepoint preserved, no token leakage, no path-safety regressions

Latent bugs surfaced (preserved per faithful-extraction contract)

1. download.py:81-86 — HEAD-skip path swallows GitPython ImportError/corruption silently (perf-only, preserved verbatim from original)
2. dry_run.py — only_packages NameError in dry-run orphan preview when lockfile exists — fixed at extraction time (helper now defaults to None)

To document in CHANGELOG before merge.

Documented technical debt

• phases/integrate.py at 861 LOC — contains 4 per-package code paths (alias / pre-downloaded / fresh download / local) that share fall-through state. Listed in KNOWN_LARGE_MODULES with 900 LOC budget. Decomposition into private _integrate_local_dep / _integrate_cached_dep / _integrate_fresh_dep helpers deferred to follow-up PR.

Follow-up PRs identified by reviewers

Item	Source	Priority
F1 — Decompose integrate.py per-package paths	architect	HIGH
F2 — Extract pipeline.py orchestrator from _install_apm_dependencies	architect	MEDIUM
F3 — Fold local-content integration in Click handler into engine pipeline	architect	LOW
F4 — UX polish (5 items: elif verbose fallbacks, security_scan tree-item method, validation/local_content logger threading, comment ASCII cleanup)	cli-logging-ux	LOW
F5 — cleanup.py:130 _targets or None widening review	security	LOW

Test results

• 3974/3974 unit tests green
• 102 targeted install/cleanup tests green
• 32 cleanup/orphan integration tests green
• 4 pre-existing failures on main confirmed unrelated

Test commands

uv run pytest tests/unit tests/test_console.py -x --no-header -q
uv run pytest tests/integration/test_intra_package_cleanup.py tests/unit/test_prune_command.py tests/unit/test_orphan_detection.py -x --no-header -q

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

Refactors the apm install implementation by extracting the install pipeline out of src/apm_cli/commands/install.py into a new src/apm_cli/install/ engine package with phase modules and a shared InstallContext.

Changes:

• Introduces apm_cli/install/ engine package (context + phases for resolve/targets/download/integrate/cleanup/lockfile/finalize).
• Thins commands/install.py into an adapter that re-exports key helpers and orchestrates the engine phases.
• Updates tests to align with the refactor (including new architecture invariants test and updated patch targets).

Show a summary per file

File	Description
tests/unit/test_install_command.py	Updates downloader patch target used by unit tests.
tests/unit/integration/test_cleanup_helper.py	Points invariant/marker test at new cleanup phase module source.
tests/unit/install/test_architecture_invariants.py	Adds LOC-budget invariant tests for the new install engine modules.
tests/unit/install/init.py	Adds unit test package marker for install tests.
tests/integration/test_selective_install_mcp.py	Updates downloader patch target used by integration tests.
src/apm_cli/install/init.py	Adds top-level documentation for the new install engine package.
src/apm_cli/install/context.py	Introduces InstallContext dataclass shared across phases.
src/apm_cli/install/validation.py	Extracts manifest/package validation helpers from the legacy install module.
src/apm_cli/install/helpers/init.py	Declares install helpers package.
src/apm_cli/install/helpers/security_scan.py	Extracts pre-deploy security scan helper.
src/apm_cli/install/presentation/init.py	Declares install presentation package.
src/apm_cli/install/presentation/dry_run.py	Extracts --dry-run rendering logic into a presentation module.
src/apm_cli/install/phases/init.py	Declares phases package.
src/apm_cli/install/phases/resolve.py	Implements lockfile loading + auth + dependency resolution + --only filtering.
src/apm_cli/install/phases/targets.py	Implements target detection and integrator initialization.
src/apm_cli/install/phases/download.py	Implements parallel pre-download via ThreadPoolExecutor.
src/apm_cli/install/phases/integrate.py	Implements sequential per-package integration loop and root .apm/ integration.
src/apm_cli/install/phases/cleanup.py	Implements orphan + stale cleanup routed through the security chokepoint helper.
src/apm_cli/install/phases/lockfile.py	Implements lockfile building/writing and deployed-file hash computation.
src/apm_cli/install/phases/finalize.py	Implements final stats rendering + fallback success output + result construction.
src/apm_cli/commands/install.py	Re-exports extracted helpers and orchestrates the new install engine phases.

Copilot's findings

• Files reviewed: 21/22 changed files
• Comments generated: 6

[danielmeppiel]

Follow-ups F1-F5 added (commits 2f82f2b..0e649ff)

All five architectural follow-ups identified by the reviewers have been addressed in this PR rather than deferred:

ID	What	Result
F1	Decompose integrate.py per-package paths	5 helpers extracted (_resolve_download_strategy, _integrate_local_dep, _integrate_cached_dep, _integrate_fresh_dep, _integrate_root_project); run() reduced from 824 to 143 LOC
F2	Extract pipeline.py orchestrator	New apm_cli/install/pipeline.py (306 LOC); commands/install.py 1268 -> 1072
F3	Fold local-content integration into pipeline	New phases/post_deps_local.py (122 LOC); eliminated duplicate target resolution + integrator init from Click handler; commands/install.py 1072 -> 933
F4	UX polish (5 items: logger plumbing, tree-item helper, ASCII cleanup)	All 5 sub-items landed
F5	Document _targets or None fallback in cleanup phase	Comment added

Final size

commands/install.py: 2905 -> 933 LOC (68% reduction)

Bonus bug-fix surfaced by F3

F3 fixed a latent bug in local .apm/ stale-cleanup: the lockfile was being read after regeneration (which doesn't write local_deployed_file_hashes), so the user-edit safety gate received empty hashes and was silently skipped. Now reads ctx.existing_lockfile (pre-install snapshot). Documented in CHANGELOG.

Re-review (3 agents in parallel)

• Architect: APPROVE WITH FOLLOW-UPS (CHANGELOG entry added; further integrate.py decomposition deferred since it's now back at 1013/1020 budget after F3 added _integrate_root_project rewrite)
• CLI logging UX: APPROVE WITH FOLLOW-UPS (2 logger-guard inconsistencies found and fixed in commit 3914f69)
• Security: APPROVE — chokepoint preserved across all new commits, F3 closes a real security gap

Tests

• 3974/3974 unit tests pass
• 32/32 integration tests pass
• 4/4 architectural invariant tests pass
• Zero non-ASCII characters across src/apm_cli/install/ and commands/install.py

[Copilot]

Copilot's findings

• Files reviewed: 30/32 changed files
• Comments generated: 9

[Copilot]

apm_modules_dir = ctx.apm_modules_dir is assigned but never used. Drop the unused local (or use it below) to avoid implying later phases still depend on bare-name locals here.

Suggested change

apm_modules_dir = ctx.apm_modules_dir

[Copilot]

These locals (package_deployed_files, package_types, _package_hashes) are initialized but never read or written back into ctx. Since later phases use ctx.package_* fields, this looks like leftover legacy scaffolding; remove them or explicitly bind them onto ctx if they are intended to be the shared containers.

Suggested change

	package_deployed_files: builtins.dict = {} # dep_key -> list of relative deployed paths
	package_types: builtins.dict = {} # dep_key -> package type string
	_package_hashes: builtins.dict = {} # dep_key -> sha256 hash
	ctx.package_deployed_files = {} # dep_key -> list of relative deployed paths
	ctx.package_types = {} # dep_key -> package type string
	ctx.package_hashes = {} # dep_key -> sha256 hash

[Copilot]

Materialization.package_info is annotated as PackageInfo, but multiple sources construct Materialization(package_info=None, ...) (see cached/fresh paths). Make this field Optional[PackageInfo] so the type contract matches actual usage and you can remove the type: ignore[arg-type] call sites.

[Copilot]

The module docstring lists options.py / InstallOptions, but there is no options.py under apm_cli/install/ in this PR. Update the architecture list to reflect the actual files (or add the missing module) so new contributors aren't sent to a dead reference.

Suggested change

options.py InstallOptions dataclass (parsed CLI options)

[Copilot]

This changelog entry does not follow the repo's Keep-a-Changelog convention of ending each bullet with a PR reference like (#764). Add the PR number (and consider splitting this into shorter, single-focus bullets) so the Unreleased section stays consistent.

Suggested change

- Refactor `apm install` internals to apply real design patterns: introduce a `DependencySource` Strategy hierarchy with shared `run_integration_template()` Template Method (kills ~300 LOC duplication across local/cached/fresh dep handlers), add `services.py` DI seam to eliminate `_install_mod` indirection, and wrap the pipeline in a typed `InstallService` Application Service consuming a frozen `InstallRequest`. `install/phases/integrate.py` shrinks from 1013 to ~400 LOC; the public `apm install` behaviour and CLI surface are unchanged. Backward-compatible: `_install_apm_dependencies` re-export and 55 healthy test patches keep working

- Refactor `apm install` internals to apply real design patterns: introduce a `DependencySource` Strategy hierarchy with shared `run_integration_template()` Template Method (kills ~300 LOC duplication across local/cached/fresh dep handlers), add `services.py` DI seam to eliminate `_install_mod` indirection, and wrap the pipeline in a typed `InstallService` Application Service consuming a frozen `InstallRequest`. `install/phases/integrate.py` shrinks from 1013 to ~400 LOC; the public `apm install` behaviour and CLI surface are unchanged. Backward-compatible: `_install_apm_dependencies` re-export and 55 healthy test patches keep working (#764)

[Copilot]

The module docstring says _integrate_local_content "remains in apm_cli.commands.install", but in this refactor the implementation lives in apm_cli.install.services (with re-exports from commands/install.py). Please update/remove this note so it reflects the current structure and patching story.

Suggested change

	The orchestrator ``_integrate_local_content`` remains in
	``apm_cli.commands.install`` because it calls ``_integrate_package_primitives``
	via bare-name lookup, and tests patch
	``apm_cli.commands.install._integrate_package_primitives`` to intercept that
	call. Keeping the orchestrator co-located with the re-exported name preserves
	``@patch`` compatibility without any test modifications.
	The orchestration entrypoint for local-content integration now lives in
	``apm_cli.install.services``. ``apm_cli.commands.install`` may re-export
	that functionality for compatibility, so tests should patch the symbol at the
	import path used by the code under test rather than assuming the
	implementation lives in the commands module.

[Copilot]

compute_package_hash as _compute_hash is imported but not used in this module. Removing the unused import will reduce noise in the pipeline seam block (and avoid suggesting hashes are still managed here).

Suggested change

from ..utils.content_hash import compute_package_hash as _compute_hash

[Copilot]

verbose_log is accepted by _local_path_no_markers_hint(...) but never used. Either remove the parameter (and the argument passed from _validate_package_exists) or use it for consistent verbose output to avoid a misleading/unused API surface.

[danielmeppiel]

@copilot address the PR review comments as needed

[Copilot]

@copilot address the PR review comments as needed

Addressed in 4656c0b:

• pipeline.py: dropped unused apm_modules_dir, package_deployed_files/package_types/_package_hashes locals, and the unused compute_package_hash as _compute_hash import
• sources.py: Materialization.package_info is now Optional[PackageInfo]; both # type: ignore[arg-type] markers removed
• install/__init__.py: refreshed the architecture docstring to match actual files (dropped dead options.py reference, added request.py/service.py/services.py/sources.py/template.py)
• phases/local_content.py: corrected docstring -- the orchestrator now lives in apm_cli.install.services and is re-exported from commands/install.py
• validation.py: removed unused verbose_log parameter from _local_path_no_markers_hint and updated the call site
• CHANGELOG.md: added (#764) to the design-patterns entry

Skipped the CHANGELOG:32 suggestion -- it's already resolved (and that bullet covers a distinct concern from the design-patterns one).

3981/3981 unit tests pass.