Skip to content

Add MaxStaleness to Support Query Acceleration and Object Tables#8509

Merged
rileykarson merged 24 commits into
GoogleCloudPlatform:mainfrom
nevzheng:biglake-query-acceleration
Aug 3, 2023
Merged

Add MaxStaleness to Support Query Acceleration and Object Tables#8509
rileykarson merged 24 commits into
GoogleCloudPlatform:mainfrom
nevzheng:biglake-query-acceleration

Conversation

@nevzheng

@nevzheng nevzheng commented Aug 1, 2023

Copy link
Copy Markdown
Contributor

The max_staleness field is used by both External Table Query Acceleration and Object Tables.

Query Acceleration requires the max_staleness field and this PR enables using Terraform to CRUD tables with query acceleration.

A prior contributor added Object Table Support. This PR extends the tests added by including max_staleness.

Changes

  • Add max_staleness field: godoc link bq rest api link
  • Update: Extended Object Table Tests w/ max_staleness
  • Add: Test demonstrating External Table with Query Acceleration

If this PR is for Terraform, I acknowledge that I have:

  • Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
  • Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
  • Generated Terraform providers, and ran make test and make lint in the generated providers to ensure it passes unit and linter tests.
  • Ran relevant acceptance tests using my own Google Cloud project and credentials (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
  • Read Write release notes before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

bigquery: added `max_staleness` to `google_bigquery_table`

@modular-magician

Copy link
Copy Markdown
Collaborator

Hello! I am a robot. It looks like you are a community contributor. @rileykarson, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

@nevzheng nevzheng changed the title Add MaxStaleness, Supporting Query Acceleration and Object Tables Add MaxStaleness to Support Query Acceleration and Object Tables Aug 1, 2023
@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 94 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 2 files changed, 94 insertions(+), 7 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 13
Passed tests 13
Skipped tests: 0
Affected tests: 0

$\textcolor{red}{\textsf{Errors occurred during REPLAYING mode. Please fix them to complete your PR}}$
View the build log

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 94 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 2 files changed, 94 insertions(+), 7 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@nevzheng

nevzheng commented Aug 1, 2023

Copy link
Copy Markdown
Contributor Author

currently running / fixing tests...

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 95 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 2 files changed, 95 insertions(+), 7 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 13
Passed tests 13
Skipped tests: 0
Affected tests: 0

$\textcolor{red}{\textsf{Errors occurred during REPLAYING mode. Please fix them to complete your PR}}$
View the build log

@rileykarson

Copy link
Copy Markdown
Member

I'm seeing the following on latest revision! Error: google-beta/resource_bigquery_table_test.go:1633:9: fmt.Sprintf format %s reads arg #7, but call has 6 args

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 91 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 2 files changed, 91 insertions(+), 7 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 2913
Passed tests 2608
Skipped tests: 302
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccBigQueryExternalDataTable_objectTable|TestAccBigQueryExternalDataTable_queryAcceleration

Get to know how VCR tests work

@modular-magician

Copy link
Copy Markdown
Collaborator

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccBigQueryExternalDataTable_objectTable[Error message] [Debug log]
TestAccBigQueryExternalDataTable_queryAcceleration[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

@rileykarson

Copy link
Copy Markdown
Member

Can you access the failed test logs? They're a little noisy, but Error: googleapi will show the issues we're running into in each.

@nevzheng

nevzheng commented Aug 1, 2023

Copy link
Copy Markdown
Contributor Author

I've managed to reproduce the issues locally and will try to fix them before uploading again. 👍

@nevzheng

nevzheng commented Aug 2, 2023

Copy link
Copy Markdown
Contributor Author

checking to see the effect of my changes

query acceleration may still need some work

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 114 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 2 files changed, 114 insertions(+), 7 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 2918
Passed tests 2614
Skipped tests: 302
Affected tests: 2

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccBigQueryExternalDataTable_objectTable|TestAccBigQueryExternalDataTable_queryAcceleration

Get to know how VCR tests work

@modular-magician

Copy link
Copy Markdown
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBigQueryExternalDataTable_objectTable[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccBigQueryExternalDataTable_queryAcceleration[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 120 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 3 files changed, 120 insertions(+), 7 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 120 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 3 files changed, 120 insertions(+), 7 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

1 similar comment
@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 120 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 3 files changed, 120 insertions(+), 7 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 2920
Passed tests 2617
Skipped tests: 302
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccBigQueryExternalDataTable_queryAcceleration

Get to know how VCR tests work

@modular-magician

Copy link
Copy Markdown
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBigQueryExternalDataTable_queryAcceleration[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{green}{\textsf{All tests passed!}}
View the build log or the debug log for each test

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 2920
Passed tests 2617
Skipped tests: 302
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccBigQueryExternalDataTable_queryAcceleration

Get to know how VCR tests work

@modular-magician

Copy link
Copy Markdown
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBigQueryExternalDataTable_queryAcceleration[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{green}{\textsf{All tests passed!}}
View the build log or the debug log for each test

@rileykarson rileykarson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants