Skip to content

Automate EEGLAB parity matrix synchronization and status reporting#241

Open
google-labs-jules[bot] wants to merge 1 commit into
developfrom
jules/sync-parity-status-js0-ba01d2a7-43a0-47a9-953b-6752fb37fef6
Open

Automate EEGLAB parity matrix synchronization and status reporting#241
google-labs-jules[bot] wants to merge 1 commit into
developfrom
jules/sync-parity-status-js0-ba01d2a7-43a0-47a9-953b-6752fb37fef6

Conversation

@google-labs-jules

Copy link
Copy Markdown

Overview

This PR introduces an automated synchronization layer between our CI test execution and the public-facing Parity Matrix documentation. Historically, the parity matrix and reference data were manually updated, leading to "documentation drift" where the reported status of numerical alignment did not always match the actual state of the codebase.

The goal of this change is to provide real-time, high-fidelity reporting of EEGLAB parity status while establishing a strict regression guardrail for numerical precision.

Key Changes

1. Statistical Metrics Collection

The compare_eeg utility and specific parity tests (e.g., test_eeg_interp.py) have been updated to extract and persist statistical tracking metrics.

  • What: Captures RMS, Mean, and Max differences.
  • Why: Moving beyond a binary pass/fail allows us to track numerical drift over time. These metrics are stored in a transient .parity_metrics.json file during the test run.

2. Parity Synchronization Tooling

A new script, tools/sync_parity_status.py, serves as the bridge between test outputs and the documentation JSON files.

  • Logic: It parses the metrics and evaluates them against configurable thresholds defined within the parity matrix files.
  • State Machine: It manages transitions between Verified Parity, In Progress, and Regressed.
  • Constraint Adherence: The script is designed to only modify value fields, preserving the existing JSON structure to ensure compatibility with our static site documentation generators.

3. Automated Regression Guarding

The synchronization script is integrated into the .github/workflows/test.yml pipeline.

  • Why: If a code change causes numerical results to deviate beyond the allowed threshold, the script returns a non-zero exit code. This fails the build, effectively preventing "float32" or precision regressions from being merged into protected branches.

4. CI/CD Integration

The workflow now includes a post-test step that:

  1. Executes the sync script on protected branches (e.g., master).
  2. Automatically commits and pushes the updated JSON files back to the repository.
  3. Ensures that the public documentation site always reflects the ground truth of the latest successful CI run.

Rationale

By automating this process, we eliminate the manual effort required per release cycle and ensure 100% consistency between internal test results and public claims. The inclusion of configurable per-function thresholds acknowledges that some algorithms are inherently non-deterministic, allowing us to tune sensitivity without triggering false positives.

Acceptance Criteria Progress

  • CI updates sample parity JSON files after successful test runs.
  • Build fails if numerical deviation exceeds thresholds.
  • Statistical metadata (RMS/Max) is correctly captured.
  • Compatible with existing S3-based data fetching.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants