Skip to content

fix(floodwait): handle re-entrant invokes instead of deadlocking - #773

Open
Davincible wants to merge 1 commit into
gotd:masterfrom
Davincible:fix/floodwait-reentrant-invoke
Open

Davincible wants to merge 1 commit into
gotd:masterfrom
Davincible:fix/floodwait-reentrant-invoke

Conversation

@Davincible

Copy link
Copy Markdown

The problem

Run's send loop calls Invoke synchronously:

for _, s := range requests {
    ret, err := w.send(s)   // blocks on the network
    ...
}

So it cannot dispatch anything else until that call returns. An invoke started
by another invoke therefore never gets sent: the send loop is blocked on the
outer call, and the outer call is blocked waiting for the nested one's result
on r.result. Both wait forever.

This is not hypothetical. gotd/td takes exactly that path on a datacenter
redirect:

upload.getFile                      -> FILE_MIGRATE_X
  telegram/invoke.go  invokeDirect  -> invokeSub(targetDC)
    telegram/pool.go  dc            -> createPool
      telegram/transfer.go transfer -> auth.exportAuthorization

exportAuth calls c.tg.AuthExportAuthorization(ctx, dcID), and c.tg is
tg.NewClient(c.invoker) — the middleware-wrapped invoker. So it comes straight
back into Handle while the send loop is still inside upload.getFile.

The symptom is not an error, it is silence. The call never returns, nothing is
logged, and a caller writing to a file leaves a zero-byte one behind. With debug
logging the trace simply stops:

Got error   err_code=303  err_msg=FILE_MIGRATE_5
telegram/invoke.go:71  Invoking on target DC  target_dc=5
telegram/pool.go:75    Creating pool  dc_id=5  max=1
<no dial, no error, no further traffic>

Every cross-datacenter download in gotd/cli hangs this way, which in practice
is most media: the file's DC frequently differs from the account's home DC.

The fix

Mark the context in send, and when Handle sees that marker, wait inline on
the calling goroutine instead of queueing.

The inline path honours the same maxWait and maxRetries bounds, so flood
waits on a nested call are still respected rather than bypassed. It deliberately
does not update per-type scheduler state: a nested call is part of an
operation already in flight, not new traffic to pace.

Scheduled traffic is completely unaffected — same queue, same throttling, same
ordering. Only the re-entrant case changes, and today that case cannot complete
at all.

Alternatives considered

  • Dispatching each send in its own goroutine. Fixes the deadlock and the
    data structures are already mutex-protected, but it changes burst behaviour
    for every user, since requests that are due would go out concurrently instead
    of one at a time. Much larger blast radius for the same bug.
  • SimpleWaiter in the caller. Works, and is what I ran locally to confirm
    the diagnosis, but it gives up global flood coordination entirely and leaves
    the underlying bug in place for everyone else.

Verification

  • New tests cover the deadlock, flood-wait retry on the nested path, and
    maxWait enforcement there. The first fails by timeout (context deadline exceeded) without this change.
  • go test ./middleware/floodwait/... passes, including -race. gofmt and
    go vet clean.
  • End to end against a real account: unmodified gotd/cli v0.11.0, still
    calling floodwait.NewWaiter(), built against this branch. An 86,425-byte
    photo on DC 5 from a home DC of 4 went from hanging indefinitely to
    downloading in 4.1 seconds. No change to gotd/cli was needed.

Run's send loop calls Invoke synchronously, so it cannot dispatch anything
else until that call returns. An invoke started by another invoke therefore
never gets sent: the send loop is blocked on the outer call, and the outer
call is blocked waiting for the nested one's result. Both wait forever.

gotd/td takes exactly that path on a datacenter redirect:

    upload.getFile                     -> FILE_MIGRATE_X
      telegram/invoke.go  invokeDirect -> invokeSub(targetDC)
        telegram/pool.go  dc           -> createPool
          telegram/transfer.go         -> auth.exportAuthorization

exportAuthorization goes through c.invoker, which is the middleware-wrapped
invoker, so it comes straight back into Handle while the send loop is still
inside upload.getFile. The symptom is not an error but silence: the call
never returns, and a caller writing to a file leaves a zero-byte one behind.
Every cross-datacenter download in gotd/cli hangs this way.

Mark the context in send, and when Handle sees that marker, wait inline on
the calling goroutine rather than queueing. The inline path honours the same
maxWait and maxRetries bounds, so flood waits are still respected. It
deliberately does not touch per-type scheduler state: a nested call is part
of an operation already in flight, not new traffic to pace.

Scheduled traffic is unaffected. Tests cover the deadlock, flood-wait retry
on the nested path, and maxWait enforcement there; the first fails by timeout
without this change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant