Optimize dags_needing_dagruns Method with Batch Processing - #37368

Closed

sunank200 wants to merge 11 commits into

apache:mainfrom

astronomer:optimize-dataset-conditional-logic

Commits on Feb 26, 2024

Enable conditional logic for datasets

dstandish
authored and
sunank200
committed
b8a6e0e
Add clear dataset fixture
sunank200
committed
6379f1f
Add example DAGs
sunank200
committed
9bec81c
Add strict in class serialization and remove example DAGs
sunank200
committed
41bd736
Query "existing dag runs" one at a time

dstandish
authored and
sunank200
committed
c1f729e
fixup! Query "existing dag runs" one at a time

dstandish
authored and
sunank200
committed
7de76eb
Enable conditional logic for datasets

dstandish
authored and
sunank200
committed
1745672
Implement | and & operators so that they can be used instead of DatasetAll and DatasetAny.
sunank200
committed
7b7389b
Add the example DAGs and fix test
sunank200
committed
250ee2b
Add documentation
sunank200
committed
0e15a3b
Optimize DAG run scheduling based on dataset triggers and batching
sunank200
committed
2f9f77f