Fix triggerer CrashLoopBackOff when json_logs is enabled - #68584
Conversation
TriggerRunnerSupervisor._process_log_messages_from_subprocess primes itself by calling airflow.sdk.log.configure_logging() with no arguments. json_output defaults to False, so this reconfigures structlog globally and installs the text WriteLogger factory -- overwriting the bytes BytesLogger factory that startup set up from json_logs=True. The stdout/stderr forwarders (_create_log_forwarder -> forward_to_log) were already wrapped with the JSON (bytes) processor chain but bind their underlying logger lazily. As soon as a trigger subprocess writes to stdout/stderr -- for example an import-time warning from a provider trigger that pulls in a heavy client (kubernetes, boto3) -- the lazy bind resolves against the now-text factory and WriteLogger.msg does `message + "\n"` on bytes from the JSON renderer, raising `TypeError: can't concat str to bytes` and crash-looping the triggerer. Pass json_output from the logging.json_logs config so the global structlog factory stays consistent with the rest of the process.
There was a problem hiding this comment.
Pull request overview
Fixes a triggerer CrashLoopBackOff that occurs when [logging] json_logs = True by ensuring the triggerer’s subprocess log-message processor reconfigures structlog with a json_output value that matches the Airflow json_logs setting (preventing a bytes-renderer/text-logger mismatch). Adds a targeted regression test and a user-facing newsfragment.
Changes:
- Pass
json_output=conf.getboolean("logging", "json_logs", fallback=False)toairflow.sdk.log.configure_logging()when priming_process_log_messages_from_subprocess. - Add a parametrized unit regression test asserting
configure_logging(json_output=<json_logs>)for bothTrueandFalse. - Add an
airflow-corebugfix newsfragment describing the triggerer crash and fix.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
airflow-core/src/airflow/jobs/triggerer_job_runner.py |
Aligns structlog reconfiguration with [logging] json_logs to prevent bytes/text logger factory mismatch during subprocess log forwarding. |
airflow-core/tests/unit/jobs/test_triggerer_job.py |
Adds a regression test ensuring the generator priming path calls configure_logging with json_output matching json_logs. |
airflow-core/newsfragments/68584.bugfix.rst |
Documents the user-visible triggerer CrashLoopBackOff bugfix. |
SameerMesiah97
left a comment
There was a problem hiding this comment.
Looks good. Just one nit.
Updated the docstring to clarify the behavior of `_process_log_messages_from_subprocess()` regarding JSON logging configuration.
hussein-awala
left a comment
There was a problem hiding this comment.
Change looks good, LGTM once the newsfragment is removed
Removed comments explaining the configure_logging function's behavior with json_logs setting.
Backport successfully created: v3-3-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
…pache#68584) * Fix triggerer CrashLoopBackOff when json_logs is enabled TriggerRunnerSupervisor._process_log_messages_from_subprocess primes itself by calling airflow.sdk.log.configure_logging() with no arguments. json_output defaults to False, so this reconfigures structlog globally and installs the text WriteLogger factory -- overwriting the bytes BytesLogger factory that startup set up from json_logs=True. The stdout/stderr forwarders (_create_log_forwarder -> forward_to_log) were already wrapped with the JSON (bytes) processor chain but bind their underlying logger lazily. As soon as a trigger subprocess writes to stdout/stderr -- for example an import-time warning from a provider trigger that pulls in a heavy client (kubernetes, boto3) -- the lazy bind resolves against the now-text factory and WriteLogger.msg does `message + "\n"` on bytes from the JSON renderer, raising `TypeError: can't concat str to bytes` and crash-looping the triggerer. Pass json_output from the logging.json_logs config so the global structlog factory stays consistent with the rest of the process. * Add newsfragment * Refactor docstring for test_process_log_messages_configures_logging Updated the docstring to clarify the behavior of `_process_log_messages_from_subprocess()` regarding JSON logging configuration. * Delete airflow-core/newsfragments/68584.bugfix.rst * Remove comments about logging configuration Removed comments explaining the configure_logging function's behavior with json_logs setting. (cherry picked from commit ebe6c58) Co-authored-by: safaehar <safae.hariri@datadoghq.com>
…pache#68584) * Fix triggerer CrashLoopBackOff when json_logs is enabled TriggerRunnerSupervisor._process_log_messages_from_subprocess primes itself by calling airflow.sdk.log.configure_logging() with no arguments. json_output defaults to False, so this reconfigures structlog globally and installs the text WriteLogger factory -- overwriting the bytes BytesLogger factory that startup set up from json_logs=True. The stdout/stderr forwarders (_create_log_forwarder -> forward_to_log) were already wrapped with the JSON (bytes) processor chain but bind their underlying logger lazily. As soon as a trigger subprocess writes to stdout/stderr -- for example an import-time warning from a provider trigger that pulls in a heavy client (kubernetes, boto3) -- the lazy bind resolves against the now-text factory and WriteLogger.msg does `message + "\n"` on bytes from the JSON renderer, raising `TypeError: can't concat str to bytes` and crash-looping the triggerer. Pass json_output from the logging.json_logs config so the global structlog factory stays consistent with the rest of the process. * Add newsfragment * Refactor docstring for test_process_log_messages_configures_logging Updated the docstring to clarify the behavior of `_process_log_messages_from_subprocess()` regarding JSON logging configuration. * Delete airflow-core/newsfragments/68584.bugfix.rst * Remove comments about logging configuration Removed comments explaining the configure_logging function's behavior with json_logs setting. (cherry picked from commit ebe6c58) Co-authored-by: safaehar <safae.hariri@datadoghq.com>
* Fix triggerer CrashLoopBackOff when json_logs is enabled TriggerRunnerSupervisor._process_log_messages_from_subprocess primes itself by calling airflow.sdk.log.configure_logging() with no arguments. json_output defaults to False, so this reconfigures structlog globally and installs the text WriteLogger factory -- overwriting the bytes BytesLogger factory that startup set up from json_logs=True. The stdout/stderr forwarders (_create_log_forwarder -> forward_to_log) were already wrapped with the JSON (bytes) processor chain but bind their underlying logger lazily. As soon as a trigger subprocess writes to stdout/stderr -- for example an import-time warning from a provider trigger that pulls in a heavy client (kubernetes, boto3) -- the lazy bind resolves against the now-text factory and WriteLogger.msg does `message + "\n"` on bytes from the JSON renderer, raising `TypeError: can't concat str to bytes` and crash-looping the triggerer. Pass json_output from the logging.json_logs config so the global structlog factory stays consistent with the rest of the process. * Add newsfragment * Refactor docstring for test_process_log_messages_configures_logging Updated the docstring to clarify the behavior of `_process_log_messages_from_subprocess()` regarding JSON logging configuration. * Delete airflow-core/newsfragments/68584.bugfix.rst * Remove comments about logging configuration Removed comments explaining the configure_logging function's behavior with json_logs setting.
What
TriggerRunnerSupervisor._process_log_messages_from_subprocessprimes itself by callingairflow.sdk.log.configure_logging()with no arguments.json_outputdefaults toFalse, so this call reconfigures structlog globally and installs the textWriteLoggerfactory — overwriting the bytesBytesLoggerfactory that startup set up when[logging] json_logs = True.The stdout/stderr forwarders (
_create_log_forwarder→forward_to_log) are already wrapped with the JSON (bytes) processor chain but bind their underlying logger lazily. As soon as a trigger subprocess writes to stdout/stderr — e.g. an import-timeDeprecationWarningfrom a provider trigger that imports a heavy client such asKubernetesPodTrigger(kubernetes) orS3KeyTrigger(boto3) — the lazy bind resolves against the now-text factory, andWriteLogger.msgdoesmessage + "\n"on bytes produced by the JSON renderer:This crashes the
TriggerRunnerSupervisor.runloop and the triggerer enters CrashLoopBackOff. It only manifests whenjson_logsis enabled and the triggerer actually runs a trigger whose subprocess emits to stdout/stderr — idle triggerers and those running only quiet triggers (DateTimeTrigger, etc.) never reachforward_to_log, which is why the crash looks intermittent across deployments.Fix
Pass
json_outputfrom the[logging] json_logsconfig (the sameconf.getboolean("logging", "json_logs", fallback=False)used elsewhere, e.g.airflow/logging_config.py) so the global structlog factory stays consistent with the rest of the process. This matches the already-hardcodedlogging_processors(json_output=True)a few lines below.Tests
Adds a parametrized regression test asserting
_process_log_messages_from_subprocesscallsconfigure_logging(json_output=<json_logs>)for bothTrueandFalse.^ Add meaningful description above. Read the Pull Request Guidelines for more information.