Skip to content

feat(go/adbc/driver/bigquery): use global backoff#145

Open
serramatutu wants to merge 4 commits into
mainfrom
serramatutu/global-backoff
Open

feat(go/adbc/driver/bigquery): use global backoff#145
serramatutu wants to merge 4 commits into
mainfrom
serramatutu/global-backoff

Conversation

@serramatutu

Copy link
Copy Markdown

The problem

dbt-labs/dbt-core#14632

The Google SDK swallows rateLimitExceeded and jobRateLimitExceeded and just keeps retrying each request with a per-request backoff. This means statements are not aware of each other, and if dbt keeps spamming the driver with new statements, they'll all use separate backoffs and hang for a long time since they're all spamming the API and consuming the rate-limit.

There is another related issue (not solved in this PR) which is that the Google SDK does not recognize that some jobs can be canceled due to rate-limits, and it keeps retrying them indefinitely. David from Columnar has already fixed this issue in the Foundry driver: adbc-drivers/bigquery#55

Changes

This PR implements a global thread-safe backoff module that can ramp up and down depending if we're getting too many rate limit requests. Each statement will use the global backoff to make its requests and update it at the end, making statement rate-limits aware of each other.

I had to switch to a fork of the Google API SDK to make it:

  1. surface the rate-limit errors so we can properly adjust the global rate-limit and
  2. allow customizeable backoffs.

I'm asking IT for permission to create a fork under the dbt Labs org, so for now the repo lives on my personal account. You can look at the 2 latest commits to see how I did that.

https://github.com/serramatutu/google-cloud-go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant