Repository navigation
test: auto-discover diff fixtures that have no registration line - #447
Merged
Merged
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
JIT quarantine entries can silently suppress auto-discovered fixtures in every execution mode.
1 open finding
What changed in this PR
Adds automatic execution of unregistered Ruby differential fixtures.
Changes:
- Discovers and runs plain fixtures in parallel.
- Adds helper exclusions and targeted fixture selection.
- Updates contributor documentation and changelog.
| File | Description |
|---|---|
crates/rubyrs/tests/diff_cruby.rs |
Implements fixture discovery and execution. |
docs/DEVELOPMENT.md |
Documents the new workflow. |
CLAUDE.md |
Updates repository testing guidance. |
CHANGELOG.md |
Records the internal change. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| .expect("read tests/diff") | ||
| .filter_map(|e| e.ok()?.file_name().into_string().ok()) | ||
| .filter_map(|f| f.strip_suffix(".rb").map(str::to_owned)) | ||
| .filter(|n| !src.contains(&format!("\"{n}\""))) |
Owner
Author
There was a problem hiding this comment.
Fixed in f3a3dda: only a call's first argument, a custom runner's name = "x", and DIFF_HELPERS count as registration. Verified with a diverging probe listed in JIT_KNOWN_DIVERGENCES: the interpreter run fails on it, the RUBYRS_JIT_NATIVE=1 run skips it.
diff_cruby.rs only ran fixtures with a hand-written #[test] line, and a forgotten line fails silently: ten fixtures (enum_for_kwargs, ivar_reflection_battery, yaml_psych_to_ruby, super_class_method_to_object and six more, some from 2026-06-27) were never run. The registration lines were also the most-touched file in the repo (102 of the last 300 commits), a merge hotspot for parallel branches. auto_discovered runs, in parallel through the same run_diff, every tests/diff/*.rb whose name this file does not mention in a string literal, so explicit registrations (ignores, cfg gates, env runners) keep working unchanged and a plain new fixture needs no line here. DIFF_HELPERS lists the require_relative/autoload helper files, and diff_helpers_exist keeps that list honest. RUBYRS_DIFF_ONLY=a,b selects fixtures the way a test filter does. All ten pass against CRuby in the interpreter, RUBYRS_JIT_NATIVE=1, tier-2 THRESHOLD=1 and STRESS_GC=1. A diverging probe fixture fails the test and is named in the message.
CLAUDE.md and DEVELOPMENT.md told contributors to add a #[test] line for every fixture. With auto_discovered that line is only needed for an attribute or a custom runner; describe that, RUBYRS_DIFF_ONLY, and DIFF_HELPERS instead.
Any quoted mention of a fixture name counted as a registration, so a
fixture added to JIT_KNOWN_DIVERGENCES would also stop running under
the interpreter and tier 2, where it must keep its coverage. Count only
a call's first argument (`("x"`), a custom runner's `name = "x"`, and
DIFF_HELPERS. Verified with a diverging probe listed in
JIT_KNOWN_DIVERGENCES: the interpreter run fails on it, the
RUBYRS_JIT_NATIVE=1 run skips it.
linyiru
force-pushed
the
test/diff-auto-discover
branch
from
October 11, 2026 04:59
b8df01e to
9e4967d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item C (first part) of the agent-native plan.
Problem.
diff_cruby.rsran only fixtures with a hand-written#[test]line. A forgotten line fails silently: ten fixtures were never run —class_self_bare_call_singleton_self,enum_for_kwargs,extend_object_override,extend_singleton_ancestors,ivar_reflection_battery,module_subclass_instance_block_call,refine_alias_primitive,super_class_method_to_object,super_in_block_forwards_method_block,yaml_psych_to_ruby(some since 2026-06-27). The registration lines also madediff_cruby.rsthe most-touched file in the repo (102 of the last 300 non-merge commits), a merge hotspot for parallel agent branches.Change. A new
auto_discoveredtest runs, in parallel through the samerun_diff(JIT_KNOWN_DIVERGENCES included), everytests/diff/*.rbwhose name the file does not mention in a string literal. Existing registrations —known:ignores,cfg/cfg_attrgates,run_diff_env, the JSON battery runner — are untouched, and a plain new fixture needs no line.DIFF_HELPERSlists the fourrequire_relative/autoloadhelper files;diff_helpers_existkeeps that list honest.RUBYRS_DIFF_ONLY=a,bselects fixtures like a test filter. No build.rs: generating tests there would recompile the rubyrs lib on every fixture edit.Verified (lawrences-mac-studio): all ten pass in the interpreter,
RUBYRS_JIT_NATIVE=1, tier-2THRESHOLD=1andSTRESS_GC=1; a diverging probe fixture (p RUBY_DESCRIPTION) fails the test and is named in the message;RUBYRS_DIFF_ONLYselects one;scripts/check.sh quickpasses.🤖 Generated with Claude Code