Skip to content

Fix triggerer CrashLoopBackOff when json_logs is enabled - #68584

Merged
vatsrahul1001 merged 5 commits into
apache:mainfrom
safaehar:fix/triggerer-json-logs-str-bytes-crash
Jul 29, 2026
Merged

Fix triggerer CrashLoopBackOff when json_logs is enabled#68584
vatsrahul1001 merged 5 commits into
apache:mainfrom
safaehar:fix/triggerer-json-logs-str-bytes-crash

Conversation

@safaehar

Copy link
Copy Markdown
Contributor

What

TriggerRunnerSupervisor._process_log_messages_from_subprocess primes itself by calling airflow.sdk.log.configure_logging() with no arguments. json_output defaults to False, so this call reconfigures structlog globally and installs the text WriteLogger factory — overwriting the bytes BytesLogger factory that startup set up when [logging] json_logs = True.

The stdout/stderr forwarders (_create_log_forwarderforward_to_log) are already wrapped with the JSON (bytes) processor chain but bind their underlying logger lazily. As soon as a trigger subprocess writes to stdout/stderr — e.g. an import-time DeprecationWarning from a provider trigger that imports a heavy client such as KubernetesPodTrigger (kubernetes) or S3KeyTrigger (boto3) — the lazy bind resolves against the now-text factory, and WriteLogger.msg does message + "\n" on bytes produced by the JSON renderer:

TypeError: can't concat str to bytes
  File ".../structlog/_output.py", line 217, in msg
    self._write(message + "\n")

This crashes the TriggerRunnerSupervisor.run loop and the triggerer enters CrashLoopBackOff. It only manifests when json_logs is enabled and the triggerer actually runs a trigger whose subprocess emits to stdout/stderr — idle triggerers and those running only quiet triggers (DateTimeTrigger, etc.) never reach forward_to_log, which is why the crash looks intermittent across deployments.

Fix

Pass json_output from the [logging] json_logs config (the same conf.getboolean("logging", "json_logs", fallback=False) used elsewhere, e.g. airflow/logging_config.py) so the global structlog factory stays consistent with the rest of the process. This matches the already-hardcoded logging_processors(json_output=True) a few lines below.

Tests

Adds a parametrized regression test asserting _process_log_messages_from_subprocess calls configure_logging(json_output=<json_logs>) for both True and False.


^ Add meaningful description above. Read the Pull Request Guidelines for more information.

TriggerRunnerSupervisor._process_log_messages_from_subprocess primes itself
by calling airflow.sdk.log.configure_logging() with no arguments. json_output
defaults to False, so this reconfigures structlog globally and installs the
text WriteLogger factory -- overwriting the bytes BytesLogger factory that
startup set up from json_logs=True.

The stdout/stderr forwarders (_create_log_forwarder -> forward_to_log) were
already wrapped with the JSON (bytes) processor chain but bind their underlying
logger lazily. As soon as a trigger subprocess writes to stdout/stderr -- for
example an import-time warning from a provider trigger that pulls in a heavy
client (kubernetes, boto3) -- the lazy bind resolves against the now-text
factory and WriteLogger.msg does `message + "\n"` on bytes from the JSON
renderer, raising `TypeError: can't concat str to bytes` and crash-looping the
triggerer.

Pass json_output from the logging.json_logs config so the global structlog
factory stays consistent with the rest of the process.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a triggerer CrashLoopBackOff that occurs when [logging] json_logs = True by ensuring the triggerer’s subprocess log-message processor reconfigures structlog with a json_output value that matches the Airflow json_logs setting (preventing a bytes-renderer/text-logger mismatch). Adds a targeted regression test and a user-facing newsfragment.

Changes:

  • Pass json_output=conf.getboolean("logging", "json_logs", fallback=False) to airflow.sdk.log.configure_logging() when priming _process_log_messages_from_subprocess.
  • Add a parametrized unit regression test asserting configure_logging(json_output=<json_logs>) for both True and False.
  • Add an airflow-core bugfix newsfragment describing the triggerer crash and fix.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
airflow-core/src/airflow/jobs/triggerer_job_runner.py Aligns structlog reconfiguration with [logging] json_logs to prevent bytes/text logger factory mismatch during subprocess log forwarding.
airflow-core/tests/unit/jobs/test_triggerer_job.py Adds a regression test ensuring the generator priming path calls configure_logging with json_output matching json_logs.
airflow-core/newsfragments/68584.bugfix.rst Documents the user-visible triggerer CrashLoopBackOff bugfix.

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just one nit.

Comment thread airflow-core/tests/unit/jobs/test_triggerer_job.py Outdated
Updated the docstring to clarify the behavior of `_process_log_messages_from_subprocess()` regarding JSON logging configuration.
@safaehar
safaehar requested a review from SameerMesiah97 June 17, 2026 08:53
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 22, 2026

@hussein-awala hussein-awala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change looks good, LGTM once the newsfragment is removed

Comment thread airflow-core/newsfragments/68584.bugfix.rst Outdated
Comment thread airflow-core/src/airflow/jobs/triggerer_job_runner.py Outdated
@hussein-awala hussein-awala added this to the Airflow 3.3.1 milestone Jul 20, 2026
@hussein-awala hussein-awala added the backport-to-v3-3-test Backport to v3-3-test label Jul 20, 2026
safaehar added 2 commits July 20, 2026 17:34
Removed comments explaining the configure_logging function's behavior with json_logs setting.
@safaehar
safaehar requested a review from hussein-awala July 20, 2026 15:36
@vatsrahul1001
vatsrahul1001 merged commit ebe6c58 into apache:main Jul 29, 2026
78 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test PR Link

github-actions Bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jul 29, 2026
…pache#68584)

* Fix triggerer CrashLoopBackOff when json_logs is enabled

TriggerRunnerSupervisor._process_log_messages_from_subprocess primes itself
by calling airflow.sdk.log.configure_logging() with no arguments. json_output
defaults to False, so this reconfigures structlog globally and installs the
text WriteLogger factory -- overwriting the bytes BytesLogger factory that
startup set up from json_logs=True.

The stdout/stderr forwarders (_create_log_forwarder -> forward_to_log) were
already wrapped with the JSON (bytes) processor chain but bind their underlying
logger lazily. As soon as a trigger subprocess writes to stdout/stderr -- for
example an import-time warning from a provider trigger that pulls in a heavy
client (kubernetes, boto3) -- the lazy bind resolves against the now-text
factory and WriteLogger.msg does `message + "\n"` on bytes from the JSON
renderer, raising `TypeError: can't concat str to bytes` and crash-looping the
triggerer.

Pass json_output from the logging.json_logs config so the global structlog
factory stays consistent with the rest of the process.

* Add newsfragment

* Refactor docstring for test_process_log_messages_configures_logging

Updated the docstring to clarify the behavior of `_process_log_messages_from_subprocess()` regarding JSON logging configuration.

* Delete airflow-core/newsfragments/68584.bugfix.rst

* Remove comments about logging configuration

Removed comments explaining the configure_logging function's behavior with json_logs setting.
(cherry picked from commit ebe6c58)

Co-authored-by: safaehar <safae.hariri@datadoghq.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jul 29, 2026
…pache#68584)

* Fix triggerer CrashLoopBackOff when json_logs is enabled

TriggerRunnerSupervisor._process_log_messages_from_subprocess primes itself
by calling airflow.sdk.log.configure_logging() with no arguments. json_output
defaults to False, so this reconfigures structlog globally and installs the
text WriteLogger factory -- overwriting the bytes BytesLogger factory that
startup set up from json_logs=True.

The stdout/stderr forwarders (_create_log_forwarder -> forward_to_log) were
already wrapped with the JSON (bytes) processor chain but bind their underlying
logger lazily. As soon as a trigger subprocess writes to stdout/stderr -- for
example an import-time warning from a provider trigger that pulls in a heavy
client (kubernetes, boto3) -- the lazy bind resolves against the now-text
factory and WriteLogger.msg does `message + "\n"` on bytes from the JSON
renderer, raising `TypeError: can't concat str to bytes` and crash-looping the
triggerer.

Pass json_output from the logging.json_logs config so the global structlog
factory stays consistent with the rest of the process.

* Add newsfragment

* Refactor docstring for test_process_log_messages_configures_logging

Updated the docstring to clarify the behavior of `_process_log_messages_from_subprocess()` regarding JSON logging configuration.

* Delete airflow-core/newsfragments/68584.bugfix.rst

* Remove comments about logging configuration

Removed comments explaining the configure_logging function's behavior with json_logs setting.
(cherry picked from commit ebe6c58)

Co-authored-by: safaehar <safae.hariri@datadoghq.com>
dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
* Fix triggerer CrashLoopBackOff when json_logs is enabled

TriggerRunnerSupervisor._process_log_messages_from_subprocess primes itself
by calling airflow.sdk.log.configure_logging() with no arguments. json_output
defaults to False, so this reconfigures structlog globally and installs the
text WriteLogger factory -- overwriting the bytes BytesLogger factory that
startup set up from json_logs=True.

The stdout/stderr forwarders (_create_log_forwarder -> forward_to_log) were
already wrapped with the JSON (bytes) processor chain but bind their underlying
logger lazily. As soon as a trigger subprocess writes to stdout/stderr -- for
example an import-time warning from a provider trigger that pulls in a heavy
client (kubernetes, boto3) -- the lazy bind resolves against the now-text
factory and WriteLogger.msg does `message + "\n"` on bytes from the JSON
renderer, raising `TypeError: can't concat str to bytes` and crash-looping the
triggerer.

Pass json_output from the logging.json_logs config so the global structlog
factory stays consistent with the rest of the process.

* Add newsfragment

* Refactor docstring for test_process_log_messages_configures_logging

Updated the docstring to clarify the behavior of `_process_log_messages_from_subprocess()` regarding JSON logging configuration.

* Delete airflow-core/newsfragments/68584.bugfix.rst

* Remove comments about logging configuration

Removed comments explaining the configure_logging function's behavior with json_logs setting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Triggerer backport-to-v3-3-test Backport to v3-3-test ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants