BigQueryHook Requires Optional Field When Parsing Results Schema

[patricker]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

apache-airflow-providers-google 8.3.0

Apache Airflow version

2.3.4

Operating System

OSX

Deployment

Virtualenv installation

Deployment details

No response

What happened

When querying BigQuery using the BigQueryHook, sometimes the following error is returned:

[2022-09-29, 19:04:57 UTC] {{taskinstance.py:1902}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/src/app/dags/test.py", line 23, in curried_bigquery
    cursor.execute(UPDATE_HIGHWATER_MARK_QUERY)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2700, in execute
    self.description = _format_schema_for_description(query_results["schema"])
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 3009, in _format_schema_for_description
    field["mode"] == "NULLABLE",
KeyError: 'mode'

What you think should happen instead

The schema should be returned without error.

How to reproduce

No response

Anything else

According to the official docs, only name and type are required to be present. mode is listed as optional.
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema

The code currently expects mode to be present.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct