Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation - #40478

Merged

Lee-W merged 51 commits into

apache:mainfrom

astronomer:dataset-alias

Jul 15, 2024

Commits on Jul 13, 2024

feat(dataset_alias): add DatasetAlias
Lee-W
committed
e7345ed
feat(datasets): add optional alias to dataset metadata
Lee-W
committed
f61be10
feat(dag): add dataset aliases defined to db
Lee-W
committed
86f6267
refactor(dag): optimize adding new dataset_aliases
Lee-W
committed
edd3b38
feat(datasets): add DatasetAlias string representation
Lee-W
committed
f6ece8f
feat(datasets): register dataset change through dataset alias in outlet event
Lee-W
committed
b89155d
feat(dataset_alias): add metadata support to datasetalias feature
Lee-W
committed
957f602
fix: fix image building issue
Lee-W
committed
e7a26ea
style: fix type hinting
Lee-W
committed
10a4d95
refactor: clean up unused code
Lee-W
committed
be5a708
test(dataset_alias): fix db_cleanup tests
Lee-W
committed
4d61a89
feat(dataset_alias): generate migration
Lee-W
committed
173246f
docs(dataset_alias): update erd
Lee-W
committed
dbbdfb3
refactor(dataset_alias): refactor OutletEventAccessor and implementation seraialization, deseralization methods
Lee-W
committed
4823554
style: fix mypy issue
Lee-W
committed
aba659c
feat(dataset_alias): generate migration
Lee-W
committed
48dce59
fix(dataset_alias): fix serialization typo
Lee-W
committed
8a4b022
refactor(dataset_alias): rename dataset_action as dataset_alias_event
Lee-W
committed
311ee31
refactor(dataset_alias): remove create_dataset_aliases and replace it with add_all
Lee-W
committed
76f51cb
refactor(datasets): remove redundant __str__ in DatasetAlias method
Lee-W
committed
ac702d5
test(datasets/metadata): add test case test_metadata_alias_handling_during_initialization
Lee-W
committed
6da2b15
test(serialization): add test cases for OutletEventAccessors
Lee-W
committed
7ccb946
test(context): add test cases to OutletEventAccessor and OutletEventAccessors
Lee-W
committed
b59fdcd
test(dataset_alias_model): add test case test_from_public
Lee-W
committed
7de3373
test(context): add test case test_add_with_db
Lee-W
committed
476208b
test(taskinstance): add test case test_outlet_dataset_alias_through_metadata and test_outlet_dataset_alias
Lee-W
committed
fcee308
feat(dag): remove removed dataset alias from db
Lee-W
committed
6ccf3ef
test(dataset_alias): add test case test_bulk_write_to_db_dataset_aliases
Lee-W
committed
b3aceb6
refactor(datasets/metadata): rename Metadata.alias as Metadata.aliases
Lee-W
committed
d5ab355
refactor(context): remove redundant OutletEventAccessor.__str__ and simplify initialization
Lee-W
committed
8e5a51d
refactor(dataset): make DatasetAlias a subclass of BaseDataset
Lee-W
committed
358761e
docs(migrations): add description to 0147 migration file
Lee-W
committed
2452e62
refactor(serialization): simplify outlet_event_accessor serialization
Lee-W
committed
f1b8dd4
refactor(datasets): merge the dataset alias handling logic to coerce_to_uri and rename it as extract_event_key
Lee-W
committed
8314403
docs(dataset_alias): add "Dynamic data events emitting through DatasetAlias" section
Lee-W
committed
a476f4f
feat(dataset_alias): change Metadata.aliases back to Metadata.aliases and forbid passing list of data alias
Lee-W
committed
c9f4b75
test(datasets.metadata): add test case test_outlet_multiple_dataset_alias
Lee-W
committed
71920c2
refactor(dataset): make DatasetAlias a subclass of BaseDataset
Lee-W
committed
6511af5
feat(dataset_alias): allow only one dataset event to the same dataset to be added in a single task
Lee-W
committed
2174693
feat(dataset_alias): raise an exception if the dataset to be aliased does not exist
Lee-W
committed
4e5c919
feat(dataset_alias): allow dynamically adding dataset through dataset_alias
Lee-W
committed
43889ca
docs(dataset_alias): update docs for dynamically created dataset
Lee-W
committed
d80201e
feat(dataset_alias): handle multiple alias add dataset should trigger only one dataset event
Lee-W
committed
e984229
refactor(dataset): make DatasetAlias a subclass of BaseDataset
Lee-W
committed
222bf22
feat(serialized_object): add dataset alias decode
Lee-W
committed
b4fe64f
docs(datasets): fix rst title format

Lee-W
and
uranusjr
committed
f919e4d
docs(dataset): fix typo

Lee-W
and
uranusjr
committed
86c074e
docs(datasets): get rid of ds in the dataset alias example
Lee-W
committed
a4e6a6f
refactor(datasets.metadata): load dataset alias as alias_name in string format
Lee-W
committed
adc830a
docs(dataset_alias): remove alembic comments
Lee-W
committed
3e238b3
docs(erd): update erd
Lee-W
committed
9b651c6