pyarrow compatibility issue when instantiating a client

[dpmcclurg]

Creating an SDK client using v2.24.1 in Databricks runtime v8.3 (Spark v3.1.1 and Python v3.8.8 and pyarrow v1.0.1) generates the following error: module 'pyarrow' has no attribute 'decimal256'.

pip version = 20.2.4

I tried:

1. Upgraded to Databricks runtime v9.0 which brings Python down to v3.8.1 and pyarrow up to v4. Same error as above and below.
2. Reverted back to Databricks runtime v8.3 (see specs above). Then, I downgraded BigQuery SDK to v2.24.0. This resolved the error.

[plamut]

@dpmcclurg Which pyarrow version do you have installed? What's the output of the following?

import pyarrow; print(pyarrow.__version__)

As of BigQuery client version v2.24.1 released last week, pyarrow >= 3.0.0 is required.

Has the pyarrow dependency been upgraded? I would check that first, and upgrade if necessary.

[tswast]

This issue has also been reported in Beam and TFX.

Closing as a duplicate of #933