Skip to content

fix: bound the daemon send path and add a readiness watchdog - #34

Open
a-whitmore-exec wants to merge 3 commits into
Endgame-Labs:mainfrom
lkosak:fix/send-path-timeouts-and-readiness
Open

fix: bound the daemon send path and add a readiness watchdog#34
a-whitmore-exec wants to merge 3 commits into
Endgame-Labs:mainfrom
lkosak:fix/send-path-timeouts-and-readiness

Conversation

@a-whitmore-exec

Copy link
Copy Markdown

Problem

A wedged outbound send could lock out the entire gateway — and it was unrecoverable remotely.

When Telegram's API became unreachable (TLS handshake timeouts to api.telegram.org), the send path had no timeout anywhere in the chain:

  1. tgbotapi's HTTP client had no timeout → bot.Send blocked forever.
  2. The daemon socket handler passed an unbounded context to the send.
  3. The goat send_user_message client had no read deadline → blocked forever waiting on the daemon.

So: the send blocked forever → the runtime (claude -p) that spawned the client blocked forever → the session never returned to idle → new inbound messages queued behind it indefinitely. The watchdog only checked kill -0 (process alive), so it saw the live-but-wedged daemon as healthy and did nothing.

Observed: two goat send_user_message processes stuck for ~2 days, holding the session open, with all message processing blocked.

Fix — three layers of defense

1. Bound the actual send (root cause)

  • telegram: give the bot a 60s HTTP client timeout — above the 30s getUpdates long-poll, below the client deadline. bot.Send ignores context, so this transport-level cap is the real fix.
  • slack: switch outbound PostMessagePostMessageContext so it honors the daemon's deadline.
  • daemon: wrap each send in a 45s context.WithTimeout.

2. Client can't hang either

  • send_user_message / send_user_file set a 90s conn.SetDeadline, so a wedged handler can't hang the helper (or the runtime blocked on it). Layered: 45s send < 60s transport < 90s client.

3. Readiness, not just liveness

  • new goated daemon status --probe does a bounded socket round-trip; exits non-zero if the daemon is down or alive-but-wedged.
  • watchdog now probes every run and force-restarts on an unresponsive socket, reaps send_user_message/send_user_file helpers older than 10 min, and takes a lock so overlapping runs don't fight during a multi-minute restart.

Testing

  • go build ./..., go vet ./..., gofmt clean; bash -n scripts/watchdog.sh clean.
  • daemon status --probe returns healthy (exit 0) against a live daemon and non-zero when down; verified backward-compatible (new probe round-trips against an older running daemon).
  • etime→seconds parser verified for mm:ss, hh:mm:ss, and dd-hh:mm:ss (macOS ps has no etimes column).

🤖 Generated with Claude Code

A wedged outbound send could lock out the whole gateway. When Telegram's
API became unreachable, the send path had no timeout anywhere in the chain:
tgbotapi's HTTP client had no timeout, the daemon socket handler passed an
unbounded context, and the goat send_user_message client had no read
deadline. The send blocked forever -> the runtime that spawned it blocked
forever -> the session never freed -> new messages queued behind it
indefinitely. The watchdog only checked `kill -0`, so it saw the live (but
wedged) daemon as healthy and did nothing. The failure was unrecoverable
remotely.

Three layers of defense:

1. Bound the actual send.
   - telegram: give the bot a 60s HTTP client timeout (above the 30s
     getUpdates long-poll, below the client deadline). bot.Send ignores
     context, so this transport-level cap is the real fix.
   - slack: switch outbound PostMessage -> PostMessageContext so it honors
     the daemon's deadline.
   - daemon: wrap each send in a 45s context.WithTimeout.

2. Client can't hang either. send_user_message / send_user_file set a 90s
   conn.SetDeadline, so a wedged handler can't hang the helper (or the
   runtime blocked on it). Layered: 45s send < 60s transport < 90s client.

3. Readiness, not just liveness.
   - new `goated daemon status --probe` does a bounded socket round-trip;
     exits non-zero if the daemon is down or alive-but-wedged.
   - watchdog probes every run and force-restarts on an unresponsive socket,
     reaps send_user_message/send_user_file helpers older than 10 min, and
     takes a lock so overlapping runs don't fight during a restart.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alan-botts

Copy link
Copy Markdown
Contributor

Blocking reliability review: the client deadlines and Telegram transport timeout are valuable, but the watchdog readiness probe does not establish that a blocked runtime/send path is healthy. The socket accepts each connection in its own goroutine, so an empty probe can succeed while the real send handler is still stuck and the serialized runtime remains blocked. Also, Block Kit Slack sends still use PostMessage rather than a context-aware call, so the new 45s daemon context does not bound that path. Please add a test that holds a real send/runtime path while the empty probe succeeds, then change the readiness signal to cover the owned work queue/runtime; and bind the Slack block paths too.

@dorkitude

Copy link
Copy Markdown
Contributor

Blocking findings addressed in 5db23de: readiness now tracks real outbound sends and reports work stuck beyond the client deadline; the regression test holds a real send while a separate socket probe runs. Slack block and thread-block sends now use PostMessageContext and have cancellation coverage. Full tests, focused race tests, vet, build, and watchdog syntax checks pass.

dorkitude added a commit to alan-botts/goated that referenced this pull request Aug 23, 2026
dorkitude added a commit to alan-botts/goated that referenced this pull request Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants