Skip to content

fix: stop Runner.__init__ exiting on an unresolvable template glob - #4966

Merged
kddejong merged 1 commit into
aws-cloudformation:mainfrom
kddejong:fix/runner-init-library-safety
Oct 30, 2026
Merged

kddejong merged 1 commit into
aws-cloudformation:mainfrom
kddejong:fix/runner-init-library-safety

Conversation

@kddejong

Copy link
Copy Markdown
Contributor

Issue

Follows #4911, which stopped schema patching from calling sys.exit(1) in a library caller's process. Two gaps remained.

Description of changes

1. Runner.__init__ still terminated the process. Four lines above the code #4911 fixed, an unresolvable --template glob was reported by calling _cli_output, which ends in sys.exit:

try:
    self.config.templates
except ValueError as e:
    self._cli_output([Match(str(e), ConfigError(), None)])

So Runner(ConfigMixIn([...])) killed the host process, and with it cfnlint.api.lint(), lint_file() and lint_by_config(). except Exception cannot catch this — SystemExit is a BaseException — so the only reliable evidence is whether a statement after the call runs. On main it does not:

$ python sentinel_runner.py        # main
E0003 /tmp/.../does-not-exist-*.yaml could not be processed by glob.glob
None:1:1
exit=2                             # "SENTINEL: host process survived" never printed
$ python sentinel_runner.py        # this branch
SENTINEL: no exception raised
SENTINEL: host process survived
exit=0

lint_by_config does reach this path: config.validate() reads the raw arguments and passes, then Runner globs them. On this branch all three API functions raise instead:

Runner.run():    TemplatesNotFoundError | CfnLintExitException=True | exit_code=2
lint_by_config:  TemplatesNotFoundError | CfnLintExitException=True | exit_code=2
lint_file:       TemplatesNotFoundError | CfnLintExitException=True | exit_code=2

with the message text unchanged. lint() no longer dies at all — it lints the string it was given, since the glob is irrelevant to it.

Why the report moved to cli() rather than main(). The error is a formatted match, not a bare log line, so --format json currently emits a valid JSON match object. Reporting it from main() (the shape #4911 used) is not possible without regressing that, because the formatters need the loaded rule set:

  • JUnitFormatter.print_matches begins if not rules: return None → --format junit would emit nothing at all
  • SARIFFormatter.print_matches does rules[rule_id].shortdesc → KeyError: 'E0003'

main() has no Runner when construction fails, so it has no rules. Instead, __init__ records the message, cli() reports it through the unchanged _cli_output, and run() raises. That keeps the CLI byte-identical by construction instead of by re-implementation, and needs no refactor of the shared exit path.

Which exception. ConfigFileError already exists and carries a config for formatting, but it is not a match: its exit code is 1 where this path's is 2, and its docstring scopes it to the .cfnlintrc file, while this is a CLI argument that resolves to no files. Reusing it would have changed the exit code for every format. TemplatesNotFoundError is a CfnLintExitException with exit code 2, matching what the CLI already returns.

2. __all__ was missing three of five subclasses. cfnlint.runner exported CfnLintExitException, InvalidRegionException and UnexpectedRuleException, but not DuplicateRuleError, ConfigFileError or SchemaPatchError. All are library-facing and that module is where a caller looks for them. All five are now exported, and a test derives the expected set from cfnlint.exceptions so a future subclass cannot be added without being exported.

CLI contract

python -c 'from cfnlint.runner import main; main()' --format <fmt> -t '<unresolvable glob>', diffed byte-for-byte against upstream/main:

scenario exit (main) exit (branch) stdout stdout bytes stderr
default 2 2 98 B identical identical
--format parseable 2 2 101 B identical identical
--format json 2 2 791 B identical identical
--format junit 2 2 46623 B identical identical
--format sarif 2 2 1740 B identical identical
--format quiet 2 2 48 B identical identical
--format pretty 1 1 0 B identical differs, see below
--non-zero-exit-code none 0 0 98 B identical identical
--output-file 2 2 0 B identical (790 B file) identical

Controls — a clean template, a template with a finding, and --list-rules — are identical in exit code and bytes.

The one deliberate difference. --format pretty on an unresolvable glob crashes with an uncaught ValueError on main and on this branch: PrettyFormatter.print_matches reads config.templates again (formatters/pretty.py:32), which raises the same error a second time. Exit code (1), stdout (empty) and the final exception are unchanged; only the traceback text differs, because the error is no longer raised from inside an except block so there is no "During handling of the above exception" chain. This is a pre-existing bug in PrettyFormatter and is left alone rather than widening this PR.

Testing

python -m pytest test                                  # 6 failed, 3276 passed

The 6 failures are the missing optional dependencies (pydot, junit-xml, sarif) and are the same set as upstream/main (6 failed, 3269 passed).

test_no_templates asserted the old behaviour — that Runner(config) itself prints and exits — with sys.exit patched to a no-op, which is what made the process kill look survivable. It now asserts that construction is silent, and that cli() produces the same bytes and the real exit code 2.

Each test was mutation-checked by reverting the change it pins:

  • revert runner/cli.py → 6 tests fail, test_no_templates with SystemExit: 2
  • revert runner/__init__.py __all__ → 5 failures
  • omit only SchemaPatchError from __all__ → still 2 failures, so the derived test is not vacuous

ruff check, ruff format --check, isort, mypy and pre-commit run --all-files all pass.

sys.exit audit

grep -rn 'sys\.exit' src/cfnlint/ returns 14 calls, all in runner/cli.py: 13 directly in cli() or main(), and one in Runner._exit, which is reached only through _cli_output, which after this change is called only from cli(). None is reachable from a constructor or from any cfnlint.api function. api.py never calls cli().


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@codecov

codecov Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.73%. Comparing base (e160ceb) to head (c80b186).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4966   +/-   ##
=======================================
  Coverage   94.73%   94.73%           
=======================================
  Files         442      442           
  Lines       16094    16102    +8     
  Branches     3123     3125    +2     
=======================================
+ Hits        15247    15255    +8     
  Misses        467      467           
  Partials      380      380           
Flag Coverage Δ
unittests 94.73% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kddejong
kddejong marked this pull request as ready for review October 30, 2026 22:35
Runner.__init__ reported an unresolvable --template glob by calling
_cli_output, which ends in sys.exit. Constructing a Runner therefore
terminated any process that embeds cfn-lint as a library, four lines above
the schema patching that aws-cloudformation#4911 fixed for the same reason. cfnlint.api
lint(), lint_file() and lint_by_config() all died the same way.

Record the error during construction and report it from cli() through the
unchanged _cli_output path, so the CLI keeps its exit code and byte-level
output for every --format, and raise TemplatesNotFoundError from run() so a
library caller gets a catchable CfnLintExitException instead of a dead
process. The report has to come from a Runner that has loaded its rules:
JUnitFormatter returns None for an empty rule set and SARIFFormatter raises
KeyError, so reporting from main() would have emitted nothing for
--format junit and crashed for --format sarif.

Also export DuplicateRuleError, ConfigFileError and SchemaPatchError from
cfnlint.runner, which already exported the other two subclasses and is
where a library caller looks for them.
@kddejong
kddejong force-pushed the fix/runner-init-library-safety branch from 3c15987 to c80b186 Compare October 30, 2026 22:36
@kddejong
kddejong merged commit a694a33 into aws-cloudformation:main Oct 30, 2026
24 checks passed
@kddejong
kddejong deleted the fix/runner-init-library-safety branch October 30, 2026 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant