Skip to content

Add BigQuery UDF support #1988

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
takbab wants to merge 28 commits into from
Closed

Add BigQuery UDF support #1988

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
b596c49
Add configuration.query.userDefinedFunctionResources to bigquery.jobs.
takbab Jul 16, 2016
39fee52
wrap to < 79 characters for PEP8
takbab Jul 16, 2016
b602d18
pylint 1.6.3 complains about not calling .
tseaver Jul 18, 2016
cdf2e37
Bump minimum pylint version to 1.6.3.
tseaver Jul 18, 2016
df47b57
Merge pull request #1990 from tseaver/pylint-1.6.3-compat
tseaver Jul 18, 2016
21ae0c9
Adding BigQuery support for legacy SQL.
dhermes Jul 18, 2016
eacccab
Fix branch coverage miss in 'gcloud.streaming.transfer'.
tseaver Jul 18, 2016
e653b95
Merge pull request #1994 from tseaver/1914-fix-branch-miss-coveralls-…
tseaver Jul 18, 2016
d171791
Merge pull request #1992 from dhermes/fix-1607
dhermes Jul 18, 2016
d3dcfcc
changed attribute name. pylint wants a max of 30 chars.
Jul 19, 2016
a54d078
PEP257 Clean-up of docstrings.
dhermes Jul 18, 2016
8ee5e03
Removing catching-non-exception from pylint disable.
dhermes Jul 19, 2016
6bd2442
Making HappyBase enabled/disabled checks into no-ops.
dhermes Jul 19, 2016
699c027
Making HappyBase compact_table() a no-op.
dhermes Jul 19, 2016
0cb6e60
Fixing language about HappyBase table compaction.
dhermes Jul 19, 2016
24c4358
Merge pull request #1997 from dhermes/fix-1989
dhermes Jul 19, 2016
08aeba1
Emulating HappyBase in counter_set.
dhermes Jul 19, 2016
22dd5e8
Updating docs for differences of HappyBase.
dhermes Jul 19, 2016
8f3469c
Merge pull request #1996 from dhermes/partial-revert-1974
dhermes Jul 19, 2016
e0addef
Merge pull request #2000 from dhermes/counter_set-happybase-just-put
dhermes Jul 19, 2016
a50936d
Fix datastore __init__ docstring.
daspecster Jul 20, 2016
b213cb8
Merge pull request #2005 from daspecster/fix-datastore-init-doc
daspecster Jul 20, 2016
9374f5b
Add configuration.query.userDefinedFunctionResources to bigquery.jobs.
takbab Jul 16, 2016
516899c
wrap to < 79 characters for PEP8
takbab Jul 16, 2016
4075bb8
changed attribute name. pylint wants a max of 30 chars.
Jul 19, 2016
34485f6
corrected.
Jul 20, 2016
dabf186
Merge remote-tracking branch 'origin/feature-bigquery-udf' into featu…
takbab Jul 20, 2016
af61528
corrected.
takbab Jul 20, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions gcloud/bigquery/job.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -869,6 +869,7 @@ class _AsyncQueryConfiguration(object):
_flatten_results = None
_priority = None
_use_query_cache = None
_user_defined_function_resources = None

This comment was marked as spam.

Sign in to view

_write_disposition = None


Expand Down Expand Up @@ -927,6 +928,12 @@ def __init__(self, name, query, client):
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query.useQueryCache
"""

user_defined_function_resources = _TypedProperty(
'user_defined_function_resources', list)
"""See:
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query.userDefinedFunctionResources
"""

This comment was marked as spam.

Sign in to view


write_disposition = WriteDisposition('write_disposition')
"""See:
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query.writeDisposition
Expand Down Expand Up @@ -965,6 +972,10 @@ def _populate_config_resource(self, configuration):
configuration['priority'] = self.priority
if self.use_query_cache is not None:
configuration['useQueryCache'] = self.use_query_cache
if self.user_defined_function_resources is not None:
configuration['userDefinedFunctionResources'] = (
self.user_defined_function_resources
)
if self.write_disposition is not None:
configuration['writeDisposition'] = self.write_disposition

Expand Down
3 changes: 3 additions & 0 deletions gcloud/bigquery/test_job.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1219,6 +1219,7 @@ class TestQueryJob(unittest2.TestCase, _Base):
JOB_TYPE = 'query'
QUERY = 'select count(*) from persons'
DESTINATION_TABLE = 'destination_table'
UDF = {"resourceUri": "gs://backet/functions.js", "inlineCode": ""}

def _getTargetClass(self):
from gcloud.bigquery.job import QueryJob
Expand Down Expand Up @@ -1420,6 +1421,7 @@ def test_begin_w_alternate_client(self):
'flattenResults': True,
'priority': 'INTERACTIVE',
'useQueryCache': True,
'userDefinedFunctionResources': [self.UDF],
'writeDisposition': 'WRITE_TRUNCATE',
}
RESOURCE['configuration']['query'] = QUERY_CONFIGURATION
Expand All @@ -1439,6 +1441,7 @@ def test_begin_w_alternate_client(self):
job.flatten_results = True
job.priority = 'INTERACTIVE'
job.use_query_cache = True
job.user_defined_function_resources = [self.UDF]
job.write_disposition = 'WRITE_TRUNCATE'

job.begin(client=client2)
Expand Down