Add a connectivity heartbeat between full speed test cycles - #23
Open
ashleigh-byte wants to merge 2 commits into
Open
Add a connectivity heartbeat between full speed test cycles#23ashleigh-byte wants to merge 2 commits into
ashleigh-byte wants to merge 2 commits into
Conversation
SLEEP_TIME (default 1800s / 30 min) means a real outage could start and fully resolve between two full speed test cycles without ever triggering the existing outage/degradation alerting, which only samples once per cycle. This adds a lightweight check in between. - Runner.ping_host(): a TCP connect to host:port, timed, returning None on failure/timeout. Uses TCP rather than ICMP specifically to avoid needing raw-socket (root) privileges this tool doesn't otherwise require. - New config: HEARTBEAT_HOST (default 1.1.1.1), HEARTBEAT_PORT (443), HEARTBEAT_INTERVAL_SECONDS (60), HEARTBEAT_CONSECUTIVE_FAILURES (3). - _wait_with_heartbeat() replaces the plain time.sleep(SLEEP_TIME) at the end of the main loop, sleeping in HEARTBEAT_INTERVAL_SECONDS chunks and pinging after each one. Alerts once after N consecutive failures, again on recovery with duration -- same alert-once-per- episode pattern as the existing outage alerting. - Deliberately single-threaded (chunked sleep, not a background thread), consistent with the rest of this codebase and avoiding any concurrency concerns with the shared sqlite connection or notifier HTTP calls. Test plan: - Unit-tested ping_host: successful TCP connect against a real local listener, refused connection, and timeout against a non-routable address (verified it respects the timeout rather than hanging) - Unit-tested HEARTBEAT_* config validation (defaults, overrides, port range boundaries) - Unit-tested _wait_with_heartbeat: chunking arithmetic (total_seconds correctly split into interval-sized chunks), exactly one down alert and one recovery alert per episode, sub-interval total_seconds producing a single chunk, and below-threshold failures never alerting
Adds a Connectivity Heartbeat section explaining the TCP-connect check and its four new env vars.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SLEEP_TIME (default 1800s / 30 min) means a real outage could start and fully resolve between two full speed test cycles without ever triggering the existing outage/degradation alerting, which only samples once per cycle. This adds a lightweight check in between.