fix: bound the daemon send path and add a readiness watchdog - #34
fix: bound the daemon send path and add a readiness watchdog#34a-whitmore-exec wants to merge 3 commits into
Conversation
A wedged outbound send could lock out the whole gateway. When Telegram's
API became unreachable, the send path had no timeout anywhere in the chain:
tgbotapi's HTTP client had no timeout, the daemon socket handler passed an
unbounded context, and the goat send_user_message client had no read
deadline. The send blocked forever -> the runtime that spawned it blocked
forever -> the session never freed -> new messages queued behind it
indefinitely. The watchdog only checked `kill -0`, so it saw the live (but
wedged) daemon as healthy and did nothing. The failure was unrecoverable
remotely.
Three layers of defense:
1. Bound the actual send.
- telegram: give the bot a 60s HTTP client timeout (above the 30s
getUpdates long-poll, below the client deadline). bot.Send ignores
context, so this transport-level cap is the real fix.
- slack: switch outbound PostMessage -> PostMessageContext so it honors
the daemon's deadline.
- daemon: wrap each send in a 45s context.WithTimeout.
2. Client can't hang either. send_user_message / send_user_file set a 90s
conn.SetDeadline, so a wedged handler can't hang the helper (or the
runtime blocked on it). Layered: 45s send < 60s transport < 90s client.
3. Readiness, not just liveness.
- new `goated daemon status --probe` does a bounded socket round-trip;
exits non-zero if the daemon is down or alive-but-wedged.
- watchdog probes every run and force-restarts on an unresponsive socket,
reaps send_user_message/send_user_file helpers older than 10 min, and
takes a lock so overlapping runs don't fight during a restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Blocking reliability review: the client deadlines and Telegram transport timeout are valuable, but the watchdog readiness probe does not establish that a blocked runtime/send path is healthy. The socket accepts each connection in its own goroutine, so an empty probe can succeed while the real send handler is still stuck and the serialized runtime remains blocked. Also, Block Kit Slack sends still use |
|
Blocking findings addressed in 5db23de: readiness now tracks real outbound sends and reports work stuck beyond the client deadline; the regression test holds a real send while a separate socket probe runs. Slack block and thread-block sends now use PostMessageContext and have cancellation coverage. Full tests, focused race tests, vet, build, and watchdog syntax checks pass. |
Problem
A wedged outbound send could lock out the entire gateway — and it was unrecoverable remotely.
When Telegram's API became unreachable (TLS handshake timeouts to
api.telegram.org), the send path had no timeout anywhere in the chain:tgbotapi's HTTP client had no timeout →bot.Sendblocked forever.contextto the send.goat send_user_messageclient had no read deadline → blocked forever waiting on the daemon.So: the send blocked forever → the runtime (
claude -p) that spawned the client blocked forever → the session never returned to idle → new inbound messages queued behind it indefinitely. The watchdog only checkedkill -0(process alive), so it saw the live-but-wedged daemon as healthy and did nothing.Observed: two
goat send_user_messageprocesses stuck for ~2 days, holding the session open, with all message processing blocked.Fix — three layers of defense
1. Bound the actual send (root cause)
getUpdateslong-poll, below the client deadline.bot.Sendignorescontext, so this transport-level cap is the real fix.PostMessage→PostMessageContextso it honors the daemon's deadline.context.WithTimeout.2. Client can't hang either
send_user_message/send_user_fileset a 90sconn.SetDeadline, so a wedged handler can't hang the helper (or the runtime blocked on it). Layered: 45s send < 60s transport < 90s client.3. Readiness, not just liveness
goated daemon status --probedoes a bounded socket round-trip; exits non-zero if the daemon is down or alive-but-wedged.send_user_message/send_user_filehelpers older than 10 min, and takes a lock so overlapping runs don't fight during a multi-minute restart.Testing
go build ./...,go vet ./...,gofmtclean;bash -n scripts/watchdog.shclean.daemon status --probereturns healthy (exit 0) against a live daemon and non-zero when down; verified backward-compatible (new probe round-trips against an older running daemon).mm:ss,hh:mm:ss, anddd-hh:mm:ss(macOSpshas noetimescolumn).🤖 Generated with Claude Code