diff --git a/README.md b/README.md index 0d5a267..378893b 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,15 @@ # EIDA Consistency Checker [![Run Tests](https://github.com/EIDA/eida-consistency/actions/workflows/test.yml/badge.svg)](https://github.com/EIDA/eida-consistency/actions/workflows/test.yml) ![Coverage](badges/coverage.svg) -[![Docs](https://img.shields.io/badge/docs-visited-blue)](https://EIDA.github.io/eida-consistency/) +[![Docs](https://img.shields.io/badge/docs-online-blue)](https://EIDA.github.io/eida-consistency/) --- A tool to evaluate the consistency between EIDA nodes' **availability** and **dataselect** web services. Designed for use in quality control and monitoring tasks across the European Integrated Data Archive (EIDA). +๐Ÿ“– **Full documentation:** + --- ## ๐Ÿš€ Installation & Quick Start @@ -57,15 +59,15 @@ Options: - `--node`: Node code (e.g. `RESIF`, `NOA`, `ETH`) - `--epochs`: Number of random test epochs (default: 10) OR percentage (e.g., `"5%"`, `0.05`) - `--duration`: Epoch length in seconds (โ‰ฅ600) -- `--seed`: Reproducible seed +- `--seed`: Random seed for sampling. Does **not** reproduce an older run โ€” to re-verify a finding see [Re-run & Re-verify](#-re-run--re-verify) - `--delete-old`: Keep only the most recent report - `--stdout`: Print JSON report to stdout -- `--report-dir`: Save reports to a custom folder (default: `reports/`) +- `--report-dir`: Save reports to a custom folder (default: `reports/`); works before or after the subcommand - `--log-level`: Control verbosity (`DEBUG`, `INFO`, `WARNING`, `ERROR`) ### Compare Reports -Compare results across two runs with the same seed: +Compare two report files (e.g. a before/after pair for the same node): ```bash uvx eida-consistency compare reports/resif_run1.json reports/resif_run2.json @@ -73,19 +75,16 @@ uvx eida-consistency compare reports/resif_run1.json reports/resif_run2.json ### Explore Inconsistencies -Drill down into inconsistencies with day-by-day exploration: +Re-check **only the inconsistencies** of a report, drilling down day-by-day to +find their exact boundaries. With no report argument it uses the newest report; +`--index` (repeatable) targets specific findings. ```bash -uvx eida-consistency explore reports/nodes/noa/*.json --index 7 --days 15 --verbose +uvx eida-consistency explore # newest report, all inconsistencies +uvx eida-consistency explore reports/noa_20260621_140111_113496.json --index 7 ``` -You can also use: - -```bash -uvx eida-consistency explore --latest -``` - -to automatically use the newest report. +See [Re-run & Re-verify](#-re-run--re-verify) for when to use this vs `check` vs a fresh run. ### Manage Node List @@ -107,8 +106,8 @@ uvx eida-consistency list-nodes Reports are stored in `./reports/` by default, or in a custom folder using `--report-dir`. -- JSON reports: `reports/resif_.json` -- Markdown reports: `reports/resif_.md` +- JSON reports: `reports/__.json` +- Markdown reports: `reports/__.md` - Global summary: [`summary.md`](https://github.com/EIDA/eida-consistency/blob/main/reports/summary.md) --- @@ -165,29 +164,53 @@ uv run mkdocs serve --- -## ๐Ÿงช Example Workflow +## ๐Ÿ” Re-run & Re-verify -### 1. Run a check for NOA: +There are three distinct ways to "run it again", depending on what you want: -```bash -uvx eida-consistency consistency --seed 1234 --node NOA --epochs 20 --duration 600 --report-dir reports/test_noa -``` -### 2. Explore incosistencies: -If inconsistencies are found: -```bash -uvx eida-consistency explore reports/nodes/resif/*.json --verbose -``` -### 3. Apply fixes -Investigate the reported service inconsistencies and fix them at node level. Maybe use dmtri command suggested. +1. **Re-verify the findings of an existing report** โ€” re-check only the + inconsistencies that a report recorded, replaying each one's exact window. + This is the right way to confirm a node-side fix: -### 4. Re-run Consistency -```bash -uvx eida-consistency consistency --seed 1234 --node NOA --epochs 20 --duration 600 --report-dir reports/test_noa -``` -### 5.Compare Before/After + ```bash + uvx eida-consistency explore reports/noa_20260621_140111_113496.json + ``` + + Add `--index N` (repeatable) to re-check specific inconsistencies only. + +2. **Re-check a single stream/window** โ€” for a one-off, targeted check: + + ```bash + uvx eida-consistency check --node NOA --net HP --sta SERG --loc "" --cha HHZ \ + --start 2016-09-20 --end 2016-10-19 + ``` + +3. **Run a fresh sampled check** โ€” draw a new random set of streams for a node: + + ```bash + uvx eida-consistency consistency --node NOA --epochs 20 --duration 600 + ``` + + > โš ๏ธ Passing the same `--seed` does **not** reproduce an older run once the + > node's live inventory changes. To reproduce a *specific* finding, use option + > 1 or 2 above, which replay the exact window rather than re-sampling. + +--- + +## ๐Ÿงช Example Workflow ```bash -uv run eida-consistency compare reports/test_noa/old.json reports/test_noa/new.json +# 1. Run a check (reports land in reports/test_noa/) +uvx eida-consistency consistency --node NOA --epochs 20 --report-dir reports/test_noa + +# 2. Re-check the inconsistencies it found +uvx eida-consistency explore --report-dir reports/test_noa + +# 3. Fix them at node level, then re-verify by replaying the same report +uvx eida-consistency explore reports/test_noa/.json + +# 4. Compare before/after +uvx eida-consistency compare reports/test_noa/old.json reports/test_noa/new.json ``` --- diff --git a/docs/cli.md b/docs/cli.md index a2c6304..d0ee67f 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -23,7 +23,13 @@ eida-consistency consistency --node NOA --epochs 10 --duration 600 * `--node TEXT`: EIDA node code (e.g., NOA, RESIF). [Required] * `--epochs INTEGER`: Number of random time to check. [Default: 10] * `--duration INTEGER`: Duration of each check in seconds (>= 600). [Default: 600] -* `--seed INTEGER`: Random seed for reproducibility. +* `--seed INTEGER`: Random seed for the candidate sampling. It does **not** + reliably reproduce an older run (a seed only selects the same streams while + the node's inventory is unchanged); to re-verify a past finding use + `explore`/`check`. See [Re-run & Re-verify](#re-run-re-verify). +* `--report-dir PATH`: Directory to store reports. Accepted before the + subcommand (`eida-consistency --report-dir DIR consistency โ€ฆ`) or after it + (`eida-consistency consistency โ€ฆ --report-dir DIR`). [Default: `reports/`] * `--upload`: Upload the report to the configured S3 bucket. ### compare @@ -36,12 +42,38 @@ eida-consistency compare report_A.json report_B.json ### explore -Explore the boundaries of inconsistent data found in a report. +Re-check **only the inconsistencies** of a report, drilling down day-by-day to +find their exact boundaries. Pass a report file, or omit it to use the newest +report in the report directory. ```bash -eida-consistency explore --index 0 +# re-check every inconsistency in the latest report +eida-consistency explore + +# re-check only specific inconsistencies in a given report +eida-consistency explore reports/noa_20260621_140111_113496.json --index 0 --index 3 ``` +**Options:** + +* `--index INTEGER`: Index of an inconsistency to explore (repeatable; default: all). +* `--days INTEGER`: Maximum days to explore backward/forward. [Default: 30] +* `--verbose`: Print query URLs while exploring. +* `--json`: Emit discovered fixes as JSON on stdout (logs stay on stderr). +* `--report-dir PATH`: Directory to load reports from (same placement rules as above). + +## Re-run & Re-verify + +There are three ways to "run it again", depending on intent: + +* **Re-verify a report's findings** โ€” replay each recorded inconsistency's exact + window: `eida-consistency explore ` (only inconsistencies; use + `--index` to target specific ones). +* **Re-check a single stream/window** โ€” `eida-consistency check --node โ€ฆ --net โ€ฆ + --sta โ€ฆ --cha โ€ฆ --start โ€ฆ --end โ€ฆ`. +* **Fresh sampled run** โ€” `eida-consistency consistency --node โ€ฆ`. Note that + `--seed` does not reproduce an older run once the node's inventory changes. + ### list-nodes List all configured EIDA nodes. diff --git a/src/eida_consistency/__init__.py b/src/eida_consistency/__init__.py index 29810b0..b3c4609 100644 --- a/src/eida_consistency/__init__.py +++ b/src/eida_consistency/__init__.py @@ -1 +1,10 @@ -"""EIDA node consistency checker package.""" \ No newline at end of file +"""EIDA node consistency checker package.""" + +from importlib.metadata import PackageNotFoundError, version as _pkg_version + +try: + __version__ = _pkg_version("eida-consistency") +except PackageNotFoundError: # pragma: no cover - package not installed + __version__ = "0.0.0+unknown" + +__all__ = ["__version__"] diff --git a/src/eida_consistency/cli.py b/src/eida_consistency/cli.py index 4e5e5ad..94c5557 100644 --- a/src/eida_consistency/cli.py +++ b/src/eida_consistency/cli.py @@ -94,10 +94,17 @@ def cli(ctx, log_level, report_dir): is_flag=True, help="Upload report to configured S3 bucket after saving locally.", ) +@click.option( + "--report-dir", + "report_dir_opt", + type=click.Path(file_okay=False, dir_okay=True, path_type=Path), + default=None, + help="Directory to store reports (overrides the global --report-dir).", +) @click.pass_context -def consistency(ctx, node, epochs, duration, seed, delete_old, print_stdout, upload): +def consistency(ctx, node, epochs, duration, seed, delete_old, print_stdout, upload, report_dir_opt): """Run availability + dataselect consistency check, or housekeeping with --delete-old.""" - report_dir: Path = ctx.obj["report_dir"] + report_dir: Path = report_dir_opt or ctx.obj["report_dir"] if delete_old: delete_old_reports(report_dir, keep=1) @@ -230,10 +237,17 @@ def check(ctx, node, net, sta, cha, loc, start, end): help="Emit the discovered fixes as JSON to stdout (machine-readable). " "Human logs/progress stay on stderr, so stdout is pure JSON.", ) +@click.option( + "--report-dir", + "report_dir_opt", + type=click.Path(file_okay=False, dir_okay=True, path_type=Path), + default=None, + help="Directory to load reports from (overrides the global --report-dir).", +) @click.pass_context -def explore(ctx, report, index, days, verbose, as_json): +def explore(ctx, report, index, days, verbose, as_json, report_dir_opt): """Explore day-by-day boundaries of inconsistencies from a report.""" - report_dir: Path = ctx.obj["report_dir"] + report_dir: Path = report_dir_opt or ctx.obj["report_dir"] if not report: try: diff --git a/src/eida_consistency/report/report.py b/src/eida_consistency/report/report.py index 8199351..23e5fbd 100644 --- a/src/eida_consistency/report/report.py +++ b/src/eida_consistency/report/report.py @@ -9,6 +9,7 @@ from pathlib import Path from typing import Any, Dict, List, Optional +from eida_consistency import __version__ from eida_consistency.core.coverage import parse_iso REPORT_DIR = Path("reports") @@ -182,6 +183,7 @@ def create_report_object( return { "summary": { + "version": __version__, "node": node, "seed": seed, "epochs_requested": epochs, @@ -285,6 +287,7 @@ def save_report_markdown(report: Dict[str, Any], report_dir: Path = REPORT_DIR) "", "## Run Summary", "", + f"- Tool version: `{summary.get('version', '?')}`", f"- Seed: `{summary['seed']}`", f"- Time: `{summary['timestamp']}`", f"- Epochs requested: `{summary['epochs_requested']}`", diff --git a/tests/report/test_report.py b/tests/report/test_report.py index 8652677..3b024bc 100644 --- a/tests/report/test_report.py +++ b/tests/report/test_report.py @@ -60,6 +60,15 @@ def test_create_report_object_basic(): assert isinstance(summary["timestamp"], str) +def test_create_report_object_includes_tool_version(): + import eida_consistency + + rep = report.create_report_object("NODE", 1, 1, 600, []) + assert rep["summary"]["version"] == eida_consistency.__version__ + assert isinstance(rep["summary"]["version"], str) + assert rep["summary"]["version"] + + def test_create_report_object_empty_records(): rep = report.create_report_object("NODE", 1, 1, 600, []) assert rep["summary"]["score"] == 0.0 @@ -98,6 +107,8 @@ def test_save_report_markdown_with_skipped(tmp_path): assert "Quality Breakdown" in text assert "Service/Network Errors: `1`" in text assert "Scored checks" in text + import eida_consistency + assert f"Tool version: `{eida_consistency.__version__}`" in text assert "Skipped checks" in text assert "TransientDataselectFailure" in text assert "| Channel | Window (UTC) | Mismatch (UTC) | Gap | Disagreement |" in text diff --git a/tests/test_cli.py b/tests/test_cli.py index df22030..d137177 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -123,6 +123,21 @@ def fake_run(**kwargs): called.update(kwargs) assert called["node"] == "NOA" +def test_consistency_report_dir_after_subcommand(monkeypatch, tmp_path): + """--report-dir placed AFTER the subcommand overrides the group default.""" + target = tmp_path / "custom" + called = {} + monkeypatch.setattr(cli, "run_consistency_check", lambda **kw: called.update(kw)) + runner = CliRunner() + result = runner.invoke( + cli.consistency, + ["--node", "NOA", "--report-dir", str(target)], + obj={"report_dir": tmp_path}, + ) + assert result.exit_code == 0, result.output + assert called["report_dir"] == target + + # ----------------- # compare command # ----------------- @@ -162,6 +177,23 @@ def test_explore_with_latest(monkeypatch, tmp_path): assert result.exit_code == 0 assert called["report"].name == "rep.json" +def test_explore_report_dir_after_subcommand(monkeypatch, tmp_path): + """explore honors --report-dir placed after the subcommand when finding latest.""" + custom = tmp_path / "custom" + custom.mkdir() + (custom / "rep.json").write_text("{}") + called = {} + monkeypatch.setattr(cli, "explore_boundaries", + lambda report, indices, max_days, verbose: called.update({"report": report})) + runner = CliRunner() + # group default (tmp_path) has no reports; the override (custom) does. + result = runner.invoke(cli.explore, ["--report-dir", str(custom)], + obj={"report_dir": tmp_path}) + assert result.exit_code == 0, result.output + assert called["report"].name == "rep.json" + assert called["report"].parent == custom + + def test_explore_no_reports(tmp_path): runner = CliRunner() result = runner.invoke(cli.explore, [], obj={"report_dir": tmp_path})