Skip to content

Simplify container log parsing and handle timestamped JSON - #1243

Merged
mmguero merged 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/765-simplify-service-log-parsing-20261015
Oct 15, 2026
Merged

mmguero merged 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/765-simplify-service-log-parsing-20261015

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor

Description

Refactor the container-log formatter to isolate JSON timestamp extraction, Supercronic event rendering, Filebeat event rendering, and per-service routing. This reduces ProcessLogLine cyclomatic complexity from F (41) to B (9) as measured by Radon, while retaining its existing formatting and sanitization behaviour.

The refactor also fixes a reproducible exception: when a container log includes an ISO-8601 timestamp before a message, the original code calls messageStrSplit[1:].join(' '). Lists do not have a join method, so timestamped JSON and timestamped plain-text lines crash. Use partition(' ') to extract the timestamp safely, preserving the remaining text. A JSON timestamp field continues to take precedence over the outer timestamp.

Related issue(s)

Refs #765 (cyclomatic-complexity reduction in Malcolm's Python scripts). The timestamp parsing bug is fixed in the same localized function.

Type of change

  • Bug fix
  • New feature or enhancement
  • Documentation
  • Other: targeted readability / complexity refactor

How has this been tested?

python -m unittest scripts.installer.tests.unit.test_process_log_line -v
radon cc -s scripts/malcolm_common.py
python -m py_compile scripts/malcolm_common.py scripts/installer/tests/unit/test_process_log_line.py

13 focused tests passed, including four cases that fail on unchanged upstream main due to the timestamped-line exception. Other tests cover ordinary JSON and plain-text logs, ISO timestamp handling and precedence, Filebeat summaries and noisy-key removal, Dashboards rendering, Supercronic success/start suppression, debug output, job errors and credential redaction.

Radon reports ProcessLogLine 41/F → 9/B, with each extracted helper rated A or B. Both files pass diff checks; the new tests pass Ruff correctness lint. The changed source passes targeted Ruff checks excluding unrelated, pre-existing import/unused-variable and bare-exception findings elsewhere in malcolm_common.py.

The wider installer unit suite was run locally: 72 tests, 4 failures and 6 errors. Those same 4 failures and 6 errors occur on unmodified upstream main (59 tests) in unrelated configuration/validation suites; no additional failures were introduced. Real container deployments, PCAP ingest, and end-to-end log pipelines were not exercised.

Documentation

  • No documentation change needed for an internal log formatter and regression tests.

Was AI used in writing the code, tests, or description for this pull request?

  • No
  • Yes (please describe below)

ChatGPT assisted with identifying the bug, refactoring the formatter, writing the regression cases and drafting this description. I inspected the result and ran the tests and checks described above.

Additional context

No new dependencies or configuration options are introduced. The existing log redaction and ignore filtering are preserved.

@mmguero mmguero self-assigned this Oct 15, 2026
@mmguero mmguero added the control.py Related to control.py script label Oct 15, 2026
@mmguero mmguero added this to the v26.10.0 milestone Oct 15, 2026
@mmguero mmguero added this to Malcolm Oct 15, 2026
@mmguero mmguero moved this to Review in Malcolm Oct 15, 2026
@mmguero
mmguero changed the base branch from main to contributor-prs October 15, 2026 19:46
@mmguero
mmguero merged commit 0915ca3 into cisagov:contributor-prs Oct 15, 2026
@github-project-automation github-project-automation Bot moved this from Review to Done in Malcolm Oct 15, 2026
@mmguero mmguero mentioned this pull request Oct 29, 2026
@mmguero mmguero moved this from Done to Released in Malcolm Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

control.py Related to control.py script

Projects

Status: Released

Development

Successfully merging this pull request may close these issues.

2 participants