Outage degradation alerting - #20
Open
ashleigh-byte wants to merge 18 commits into
Open
Conversation
Pure schema change with no consumers yet: nmap-derived MAC address, vendor, and hostname data will be tracked per device scan starting in a follow-up PR. Existing databases auto-migrate in place via _migrate_device_scans_columns() so upgrading doesn't require recreating the database.
…ty tracking Builds on the device_scans schema from #<PR1>. Extracts nmap-derived MAC address, vendor, and hostname per device (nmap only resolves MAC/vendor for hosts on the same local subnet it can ARP directly, which the existing sudo ARP scan already covers). Adds get_latest_devices_with_novelty(), which flags devices whose MAC hasn't been seen in any scan within the last 14 days -- devices with no resolvable MAC are never flagged as new, since there's no reliable identity to compare against. No reporting/AI changes yet -- this is data layer only. Surfacing this in the AI report is a separate follow-up PR.
Combines PR #3 (Device Watch reporting) and #4 (AI report restructuring) since #3's third JSON field only makes sense on top of #4's restructured prompt. AI restructuring: the AI now returns only 3 short JSON fields (dynamics_analysis, device_watch, conclusion) instead of a full literal HTML template. Several capable local models (llama3.1:8b, qwen2.5:7b-instruct) reliably abandoned the old rigid template under load and fell back to generic assistant-style summaries; JSON output + code-assembled HTML (REPORT_TEMPLATE_SHELL) guarantees correct, consistent formatting regardless of which model is behind AI_BASE_URL. Malformed or missing JSON falls back to plain non-AI text per field rather than losing the whole report -- unchanged from the existing "AI failures degrade gracefully" behavior. Device Watch reporting: builds on the data layer from #2 -- a vendor-count breakdown of currently-online devices plus a short new-devices list (MACs not seen in the last 14 days) is now included in the AI's user message, so the "device_watch" field can call out anything genuinely new without being fed noisy per-IP detail for every device. Not included here: notifier truncation fixes, alert-then-crash on infra failures, and misc config additions (AI_CONTEXT_SIZE, SLEEP_TIME/ REPORT_CYCLE_COUNT, --test-ai, temperature) -- those are separate PRs.
…n logs Speed test, device scan, database, and notifier-delivery failures still crash the process (existing project philosophy: don't silently retry forever), but now make a best-effort attempt to post an alert to the configured notifier first, so the failure is visible without checking server logs. A failure to send that alert itself is logged separately and never masks the original exception. Validated against real production incidents: a missing sudoers grant, a disk-full DB write, and a DNS outage that took down both the speed test and the notifier webhook at once -- in all three cases the alert attempt fails alongside the primary error, and the original exception still propagates and crashes the process rather than being swallowed. The existing AI-backend-failure handling (falls back to plain text, does not crash) is untouched -- that's a deliberately different philosophy: an AI hiccup degrades gracefully, an infra failure is a real bug in netmon and should be loud.
…st-ai, and bump AI temperature - SLEEP_TIME and REPORT_CYCLE_COUNT were previously hardcoded (1800s, 8 cycles); now configurable via env vars with the same defaults, so no behavior changes for anyone who doesn't set them. - Added optional AI_CONTEXT_SIZE, passed through to Ollama as num_ctx via extra_body. Local models' default context window (often 2048-4096 tokens) can silently truncate a long system prompt + a day of history, dropping persona/format instructions while keeping raw data -- this looks identical to a model-capability failure but isn't one. No-op on cloud OpenAI since it's only sent when explicitly configured. - Added --test-ai flag to force one detailed-report cycle immediately (AI + graph + notifier delivery), then auto-resume the normal schedule with nothing to remember to revert afterward. - Bumped AI temperature to 0.9 (previously left at the API default) for more personality in commentary, overridable per-call.
Only the optional Ookla CLI backend (install-ookla-speedtest.sh) reports jitter and packet loss; classic speedtest-cli has no equivalent fields, so this stays inactive-by-default and additive -- nothing changes for anyone not using that backend. - metrics table gains two nullable columns, jitter_ms and packet_loss_pct (auto-migrated in place, same pattern as the recent device_scans migration) - NetworkMetric gains matching optional fields, populated from the Ookla JSON output when present, left None for classic speedtest-cli - Surfaced in mini reports, detailed reports, and the AI's 24h history context via three small per-template jitter-line helpers that return an empty string when the data isn't available, so the report layout has no stray blank line or label when the fields are absent - The AI is told to treat high jitter (tens of ms) or nonzero packet loss as a bufferbloat signal distinct from raw throughput -- a connection can have great Mbps and still feel laggy under load if jitter is high -- and to say nothing about it when the data's absent Depends on the JSON-restructured report architecture (dynamics_analysis/ device_watch/conclusion fields, REPORT_TEMPLATE_SHELL) since that's where this was actually prototyped and tested -- retrofitting it onto the old literal-HTML-template prompt would be a step backward given that architecture is already known to fail under load with local models.
…tion-alerting # Conflicts: # main.py
# Conflicts: # main.py
A small state machine tracks consecutive bad or failed speed-test readings and alerts immediately when an outage/degradation is confirmed, then again once it clears (with duration) -- instead of waiting for the next scheduled detailed report, which could be up to REPORT_CYCLE_COUNT cycles away. - New config: OUTAGE_DOWNLOAD_THRESHOLD_MBPS (default 20), OUTAGE_PING_THRESHOLD_MS (default 150), OUTAGE_CONSECUTIVE_READINGS (default 2). A reading counts as "bad" if it's below the download threshold or above the ping threshold. - Alerts once per episode, not every cycle, to avoid spam; a recovery alert fires once state returns to normal, reporting how long the episode lasted and which kind of episode it was (outage vs degraded). - A speed test failing outright feeds the same "bad reading" counter as a degraded-but-successful reading, and skips the device scan/DB write/reporting entirely for that cycle (there's no metric to record). Design decision (flagged for discussion before this was written -- see the tracking issue): this deliberately treats a failing/degraded speed test as an *expected* condition -- the exact thing this tool exists to detect -- so it alerts and keeps retrying every cycle, rather than crashing the process the way the alert-then-crash PR treats infra failures (nmap/database/notifier problems). The two are mutually exclusive by construction: a speed test exception is caught and handled here before it can reach the outer crash-on-infra-failure handler. Depends on the JSON-restructured report architecture, jitter reporting, alert-then-crash, and the configurable sleep_time/report_cycle_count/ ai_context_size additions -- this sits on top of all of them, matching how it was actually prototyped and tested.
Updates the Features & Workflow overview, adds a Device Watch section explaining MAC/vendor/hostname tracking and the new-device check, and updates the example detailed report to show the Device Watch field and note that the AI only ever produces short text fields -- the surrounding HTML is assembled in code, not generated by the model.
Adds a Jitter & Bufferbloat section noting this only activates with an Ookla-compatible backend (classic speedtest-cli has no equivalent data), and updates the example detailed report to show the optional Jitter line.
Adds a Reliability section explaining that infra failures (speed test, device scan, database, notifier) post a best-effort alert then crash rather than silently retrying, and contrasts this with the existing graceful-degradation behavior for AI backend failures specifically.
… and --test-ai Updates the Features & Workflow wording (no longer implies these are hardcoded), adds the three new env vars to the .env variables table, and documents the --test-ai flag for forcing an immediate detailed report during setup/testing.
# Conflicts: # README.md
Adds an Instant Outage & Degradation Alerting section with the three new env vars, a note distinguishing this from the Reliability section's crash-on-infra-failure behavior, and a Features & Workflow bullet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A state machine tracks consecutive bad/failed speed-test readings and alerts immediately on a confirmed outage/degradation, then again on recovery with duration -- instead of waiting for the next scheduled report.
Depends on the full report/jitter/alert-then-crash/config stack (this branch's history includes those merges).