Handling “Airflow confs” in Task SDK

[kaxil]

Figure out how to retrieve Airflow configurations (from airflow.cfg) in the Task SDK without importing Airflow core.

Investigate different approaches for configuration architecture in separated server and SDK components, analyze trade-offs, and recommend the best approach.

Configuration Split Options to Investigate

Option 1: Complete Independence

• Approach: Separate configuration files for server and SDK
• Pros: Clean separation, no network dependencies
• Cons: Configuration duplication, sync issues
• Use case: When server and workers are in different environments

Option 2: API-Based Inheritance

• Approach: SDK queries server API for configuration values
• Pros: Centralized configuration management
• Cons: Network dependency, potential latency
• Use case: When workers need dynamic configuration updates

Option 3: Bootstrap Injection

• Approach: Server injects essential config at task startup
• Pros: No runtime API calls, centralized control
• Cons: Static configuration, larger task payloads
• Use case: When configuration is stable and performance is critical

Option 4: Hybrid Local-First + API Fallback

• Approach: Local config first, API fallback for inherited values, caching
• Pros: Performance + flexibility, graceful degradation
• Cons: Complexity, cache invalidation challenges
• Use case: Best of both worlds approach

## Open Questions

• What about ProvidersManager class that is used to fetch configs from Providers

[kaxil]

@potiuk 's options here: https://lists.apache.org/thread/8jtgr1m072lgsy4tl3km9ptb216vqjds

we have three
possible options to tell our users:.

1. either copy to workers only the configuration that is needed (but which
   ones are needed and how to separate those config entries in the docs?)
2. or NO configuration should be present in the worker - and all
   configuration that is needed should be passed by Task SDK to the worker
3. or we assess that task and dag file processors do not need any
   configuration at all

I think if we assess that 3) is the case - there is no other option but to
remove conf. That would also be really nice as we would not have to have
"conf" reading code in TaskSDK at all.
But if we think that tasks and dag parsing will actually need some
configuration, then we can go either 1) or 2) route

I think option 2) is much better from less confusion, security and "single
source of truth" point of view. And if we go for 2) then namespace = conf.get("kubernetes", "NAMESPACE") will not work because configuration
will not be present there and in this case what would make more sense is to
leave "conf" as part of the context but only pass the "needed"
configuration there.
If we go for option 1) then likely yes we could remove it from context.

[potiuk]

Thanks @kaxil - yeah, that's the right place to discuss it :)