FEAT : Add command to get DAG Details via CLI

[maahir22]

This PR closes #30365
Problem Statement: At the moment, there is no support to view DAG Details via the Airflow CLI. The only support for viewing DAG Details is via the REST API at the endpoint: https://airflow.apache.org/api/v1/dags/{dag_id}/details

This new command in Airflow has the proposed format:
airflow dags details <dag-id> and returns all the details that are returned from the REST API as well. The terminal output currently looks like this:

[hussein-awala]

You are trying to implement a new logic for yaml output, and this doesn't support other types (json, table, plain).
Instead you can use AirflowConsole.print_as method, something like:

Suggested change

	for key, value in dag_detail.items():
	if isinstance(value, dict):
	print(f"\t{key}:")
	for subkey, subvalue in value.items():
	print(f"\t\t{subkey}: {subvalue}")
	else:
	print(f"\t{key}: {value}")
	AirflowConsole().print_as(
	data=[dag_detail],
	output=args.output,
	)

[potiuk]

Yep. All the CLI commands should use the same formatting options.

[maahir22]

Are you sure you want to do that @potiuk, @hussein-awala ? The default fallback when no --output option is provided is table -> so args.output becomes table by default, the SimpleTable isn't able to render such a large table properly due to the number of parameters and the length of these fields - most devices won't be able to display this output in a comprehensible way.
To standardise the approach, I've switched to AirflowConsole(), with a caveat that "table" isn't allowed as an output option - if it's received "yaml" output is displayed instead.

1. Table Format

2. JSON Format
   

3. YAML Format
   

4. Plain Format
   

[potiuk]

Can you make a table support with swapped columns and rows ? I think that one will be much better.

[maahir22]

Ooo so essentially a transpose of the current table?
Let me know if I understand it currently, if initially the columns were scheduler_lock, is_active with Row1 being [], False -> now the rows will be scheduler_lock, is_active with Column1 having [], False?
Also, do we want this to be a new output type like --ttable? (Extra t for transposed)

[hussein-awala]

I like the idea of the transposed table, it's simple with no need to update the print_as method:

if args.output == "table":
    data = [
        {
            "property_name": key,
            "property_value": value
        }
        for key, value in dag_detail.items()
    ]
else:
    data = [dag_detail]

[potiuk]

precisely.

[maahir22]

Changes done, and everything looks fine now for table (all other outputs in my previous comment) -

[maahir22]

As you can see in my previous comments, table & plain both formats were not rendered properly for the large number of attributes, so I've added checks for both of them.

[potiuk]

Nice. That's what I thought will be nicer :)

[hussein-awala]

LGTM

[potiuk]

This is not the right way. You should check attributes of the dag rather than list the fields by hand. If we ad a new field we wll 100% forget to add it here.

[maahir22]

Got it, I'll check all the attributes - don't get the "they should also likely be sorted part", do we want to display the command attributes in sorted order & test if the same order is maintained? Is there a reason behind doing this as it follows the same order as the REST Client response.

[potiuk]

Needs non-hardcoded attributes, they should be retrieved from dag_details.
There are a number of ways you can do it: https://stackoverflow.com/questions/2675028/list-attributes-of-an-object
Possibly you should filter some (those starting with _ and other common object attributes. They should also likely be sorted.

[maahir22]

Needs non-hardcoded attributes, they should be retrieved from dag_details. There are a number of ways you can do it: https://stackoverflow.com/questions/2675028/list-attributes-of-an-object Possibly you should filter some (those starting with _ and other common object attributes. They should also likely be sorted.

@potiuk On closer inspection, I found that listing attributes of the class directly isn't going to be a scalable solution either - we have to omit function names / other variable names in such a case as well - highly probable that someone adds a new function to the schema and doesn't add it to the test.
So, I've simply mimicked the process -> we get the DAG by querying and then dump to obtain the schema fields, finally we just check whether all fields present in this dumped schema are present in the response obtained from the CLI Command. This should be a scalable solution since we don't have to worry about hardcoding variables/function names anywhere anymore.
OR
If this approach seems to be too duplicate, I can check if the intersection of CLI Response Fields and the dir(dag_schema) >= FIXED_THRESHOLD, this will ensure that incase new fields are added in future our check don't fail.
@josh-fell @hussein-awala Please let me know if your opinion on this aswell.
PS: In our case as per the StackOverflow link only dir worked - vars & dict threw unreliable empty results.

[maahir22]

LGTM Merge

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[potiuk]

JUST in time for 2.6