Automate EEGLAB parity matrix synchronization and status reporting#241
Open
google-labs-jules[bot] wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR introduces an automated synchronization layer between our CI test execution and the public-facing Parity Matrix documentation. Historically, the parity matrix and reference data were manually updated, leading to "documentation drift" where the reported status of numerical alignment did not always match the actual state of the codebase.
The goal of this change is to provide real-time, high-fidelity reporting of EEGLAB parity status while establishing a strict regression guardrail for numerical precision.
Key Changes
1. Statistical Metrics Collection
The
compare_eegutility and specific parity tests (e.g.,test_eeg_interp.py) have been updated to extract and persist statistical tracking metrics..parity_metrics.jsonfile during the test run.2. Parity Synchronization Tooling
A new script,
tools/sync_parity_status.py, serves as the bridge between test outputs and the documentation JSON files.Verified Parity,In Progress, andRegressed.3. Automated Regression Guarding
The synchronization script is integrated into the
.github/workflows/test.ymlpipeline.4. CI/CD Integration
The workflow now includes a post-test step that:
master).Rationale
By automating this process, we eliminate the manual effort required per release cycle and ensure 100% consistency between internal test results and public claims. The inclusion of configurable per-function thresholds acknowledges that some algorithms are inherently non-deterministic, allowing us to tune sensitivity without triggering false positives.
Acceptance Criteria Progress