feat(go/adbc/driver/bigquery): use global backoff#145
Open
serramatutu wants to merge 4 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
dbt-labs/dbt-core#14632
The Google SDK swallows
rateLimitExceededandjobRateLimitExceededand just keeps retrying each request with a per-request backoff. This means statements are not aware of each other, and if dbt keeps spamming the driver with new statements, they'll all use separate backoffs and hang for a long time since they're all spamming the API and consuming the rate-limit.There is another related issue (not solved in this PR) which is that the Google SDK does not recognize that some jobs can be canceled due to rate-limits, and it keeps retrying them indefinitely. David from Columnar has already fixed this issue in the Foundry driver: adbc-drivers/bigquery#55
Changes
This PR implements a global thread-safe backoff module that can ramp up and down depending if we're getting too many rate limit requests. Each statement will use the global backoff to make its requests and update it at the end, making statement rate-limits aware of each other.
I had to switch to a fork of the Google API SDK to make it:
I'm asking IT for permission to create a fork under the dbt Labs org, so for now the repo lives on my personal account. You can look at the 2 latest commits to see how I did that.
https://github.com/serramatutu/google-cloud-go