dekaf: brief control-plane errors break Kafka client auth despite a valid cached token

[jwhartley]

1. Priority

Medium - one tenant confirmed, recurring roughly every two days, and the longest occurrence stopped the customer's downstream pipelines and required a manual restart.

2. Problem description

• Expected: a brief control-plane hiccup does not break Kafka client authentication, because the Dekaf task manager already holds an unexpired cached token for the task.

• Observed: every Kafka session for the affected tasks fails to authenticate for as long as the control plane is returning errors. The customer's client retried for about 5 minutes, gave up, and stopped 3 downstream pipelines, which then needed a manual restart to resume.

  Client-visible error:

  Failed to look up or compile schema id <id> of record 1 from registry. Forwarded error:"REST request failed (code 500): Unknown( SharedError( Error { context: "error fetching or refreshing dekaf task auth", source: "POST /authorize/dekaf: 500 Internal Server Error: failed to fetch task spec: pool timed out while waiting for an open connection", },"
  

  Dekaf-side, 3117 lines during the 08-27 occurrence:

  ERROR ...:run_task_manager{...task_name="<task>"}: dekaf::task_manager: Error in task manager loop task_name="<task>" error=error fetching or refreshing dekaf task auth
  

  The control-plane responses were a mix of 500 and 503 across the 13 occurrences, not a single status code. Only the 08-27 occurrence carried pool timed out while waiting for an open connection; the others had empty bodies.

• Two observations from reading crates/dekaf/src/task_manager.rs at b5b98f57343, either of which would keep a cached token in play:

  1. The cached-token fallback runs only when the refresh call times out (lines 767-779). A non-2xx HTTP response takes a different arm and propagates without the cached token being consulted.
  2. The loop passes cached_dekaf_auth.take() (line 321) and only writes the value back on success (line 331), so a failed call leaves no cached token for the next iteration.

• Steps to reproduce: with a Dekaf task whose cached token is inside its refresh window, return a 500 or 503 from POST /authorize/dekaf for longer than one refresh interval, then connect a Kafka client. Authentication fails for the duration instead of being served from the cached token.

3. Scope and prevalence

One tenant, four Dekaf materialization tasks sharing a single Kafka client (SingleStore), on a private data plane.

13 occurrences in 30 days (2026-08-02 to 2026-09-01). Eleven lasted under a minute and went unnoticed. Two hit all four tasks at once:

• 2026-08-27 19:18:23Z to 19:24:26Z, 6.1 minutes. This is the one the customer reported.
• 2026-09-01 17:37:22Z to 17:38:52Z, 1.5 minutes.

4. Possible improvements

Support-authored from reading the code, not verified by running it. Recorded as starting points, not as a chosen design.

1. Serve the cached token on a non-2xx response the same way it is already served on a timeout, so a transient control-plane error is absorbed rather than surfaced to the client.
2. Keep the cached token when a refresh call fails, so a subsequent failure still has a token to fall back on.

5. References

• Related prior art: https://estuaryworkspace.slack.com/archives/C03PF2FSM5H/p1767805810312529
• Related prior art: https://estuaryworkspace.slack.com/archives/C03PF2FSM5H/p1746813698497609