refactor(hydro_test): migrate paxos cross_singleton to by_ref() singleton mechanism [ci-bench] by MingweiSamuel · Pull Request #2903 · hydro-project/hydro

MingweiSamuel · 2026-05-28T21:41:45Z

We need to remove the global singleton counter and instead use a local counter.

Converted 6 out of 7 cross_singleton usages in paxos.rs to use the new
by_ref() singleton mechanism:

recommit_after_leader_election: 3 cross_singleton calls replaced with
by_ref() captures in filter_map and map closures
sequence_payload: 1 cross_singleton replaced with by_ref() in map
index_payloads (sliced!): 1 cross_singleton replaced with by_ref() in map
acceptor_p2: 2 cross_singleton calls replaced with by_ref() captures
in filter_map and map closures

Left unchanged:

acceptor_p1: 2 cross_singleton calls remain because the a_log parameter
comes from a forward_ref cycle, and by_ref() on cycle sources causes a
graph partitioning error ("Handoff succ not in subgraph")

Also added + 'a lifetime bounds where required by by_ref() usage:

PaxosLike::build, PaxosLike::with_client
paxos_core, index_payloads
compartmentalized_paxos_core

Updated IR snapshots to reflect the new singleton ref nodes.

Co-authored-by: Infinity 🤖 infinity@hydro.run

cloudflare-workers-and-pages · 2026-05-28T21:41:47Z

Deploying hydro with Cloudflare Pages

Latest commit:	`2f9004d`
Status:	✅ Deploy successful!
Preview URL:	https://5d79373b.hydroflow.pages.dev
Branch Preview URL:	https://mingwei-paxos-ref.hydroflow.pages.dev

View logs

github-actions · 2026-05-28T21:42:07Z

📊 Benchmark Results

✅ Benchmark completed! You can download the results from the links below.

Run History:

Run #4972 - ✅ Complete (Download Artifact)
Run #4970 - ✅ Complete (Download Artifact)
Run #4951 - ✅ Complete (Download Artifact)
Run #4935 - ✅ Complete (Download Artifact)
Run #4923 - ✅ Complete (Download Artifact)
Run #4922 - ✅ Complete (Download Artifact)
Run #4764 - ✅ Complete (Download Artifact)

_{Last updated: 2026-06-10T19:50:36.699Z}

Added `with_singleton_capture` wrapper to `Stream::filter_map` so that `SingletonRef` values can be captured inside `filter_map` closures via `q!()`, matching the existing support in `map`, `filter`, `flat_map_ordered`, `flat_map_unordered`, `partition`, and `inspect`. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

…eton mechanism [ci-bench] We need to remove the global singleton counter and instead use a local counter. Converted 6 out of 7 cross_singleton usages in paxos.rs to use the new by_ref() singleton mechanism: - recommit_after_leader_election: 3 cross_singleton calls replaced with by_ref() captures in filter_map and map closures - sequence_payload: 1 cross_singleton replaced with by_ref() in map - index_payloads (sliced!): 1 cross_singleton replaced with by_ref() in map - acceptor_p2: 2 cross_singleton calls replaced with by_ref() captures in filter_map and map closures Left unchanged: - acceptor_p1: 2 cross_singleton calls remain because the a_log parameter comes from a forward_ref cycle, and by_ref() on cycle sources causes a graph partitioning error ("Handoff succ not in subgraph") Also added + 'a lifetime bounds where required by by_ref() usage: - PaxosLike::build, PaxosLike::with_client - paxos_core, index_payloads - compartmentalized_paxos_core Updated IR snapshots to reflect the new singleton ref nodes. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

Added `with_singleton_capture` wrapper to `Stream::filter_map` so that `SingletonRef` values can be captured inside `filter_map` closures via `q!()`, matching the existing support in `map`, `filter`, `flat_map_ordered`, `flat_map_unordered`, `partition`, and `inspect`. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

…eton mechanism [ci-bench] We need to remove the global singleton counter and instead use a local counter. Converted 6 out of 7 cross_singleton usages in paxos.rs to use the new by_ref() singleton mechanism: - recommit_after_leader_election: 3 cross_singleton calls replaced with by_ref() captures in filter_map and map closures - sequence_payload: 1 cross_singleton replaced with by_ref() in map - index_payloads (sliced!): 1 cross_singleton replaced with by_ref() in map - acceptor_p2: 2 cross_singleton calls replaced with by_ref() captures in filter_map and map closures Left unchanged: - acceptor_p1: 2 cross_singleton calls remain because the a_log parameter comes from a forward_ref cycle, and by_ref() on cycle sources causes a graph partitioning error ("Handoff succ not in subgraph") Also added + 'a lifetime bounds where required by by_ref() usage: - PaxosLike::build, PaxosLike::with_client - paxos_core, index_payloads - compartmentalized_paxos_core Updated IR snapshots to reflect the new singleton ref nodes. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

Added `with_singleton_capture` wrapper to `Stream::filter_map` so that `SingletonRef` values can be captured inside `filter_map` closures via `q!()`, matching the existing support in `map`, `filter`, `flat_map_ordered`, `flat_map_unordered`, `partition`, and `inspect`. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

…eton mechanism [ci-bench] We need to remove the global singleton counter and instead use a local counter. Converted 6 out of 7 cross_singleton usages in paxos.rs to use the new by_ref() singleton mechanism: - recommit_after_leader_election: 3 cross_singleton calls replaced with by_ref() captures in filter_map and map closures - sequence_payload: 1 cross_singleton replaced with by_ref() in map - index_payloads (sliced!): 1 cross_singleton replaced with by_ref() in map - acceptor_p2: 2 cross_singleton calls replaced with by_ref() captures in filter_map and map closures Left unchanged: - acceptor_p1: 2 cross_singleton calls remain because the a_log parameter comes from a forward_ref cycle, and by_ref() on cycle sources causes a graph partitioning error ("Handoff succ not in subgraph") Also added + 'a lifetime bounds where required by by_ref() usage: - PaxosLike::build, PaxosLike::with_client - paxos_core, index_payloads - compartmentalized_paxos_core Updated IR snapshots to reflect the new singleton ref nodes. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

… non-determinism The global `static SINGLETON_REF_COUNTER: AtomicUsize` was shared across all tests in a binary, causing `__hydro_singleton_ref_N` idents to shift depending on test execution order. For example, paxos snapshots expected IDs 0-6 but got 12-18 when singleton_ref tests ran first. Fix: Remove the global counter entirely. Instead, derive the ident index from `refs.len()` within each `with_singleton_capture` scope, so every closure's captured singleton refs are numbered starting from 0. This is safe because each closure gets its own lexical block in the generated code. Changes: - hydro_lang/src/singleton_ref.rs: Remove `SINGLETON_REF_COUNTER` static. Add `singleton_ref_ident(index)` helper. Simplify thread-local from `Vec<(syn::Ident, HydroNode)>` to `Vec<HydroNode>`. Use `refs.len()` as index. - hydro_lang/src/compile/ir/mod.rs: Change `ClosureExpr::singleton_refs` from `Vec<(syn::Ident, HydroNode)>` to `Vec<HydroNode>`. Update Clone, Serialize, deep_clone, transform_children, and emit_tokens to use index-based idents via the new helper. - hydro_test snapshots: Regenerated paxos IR and mermaid snapshots (stable only; nightly snapshots need a nightly CI run). Also adds test to ensure singleton references are handled in the correct order. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2929

Added `with_singleton_capture` wrapper to `Stream::filter_map` so that `SingletonRef` values can be captured inside `filter_map` closures via `q!()`, matching the existing support in `map`, `filter`, `flat_map_ordered`, `flat_map_unordered`, `partition`, and `inspect`. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

…eton mechanism [ci-bench] We need to remove the global singleton counter and instead use a local counter. Converted 6 out of 7 cross_singleton usages in paxos.rs to use the new by_ref() singleton mechanism: - recommit_after_leader_election: 3 cross_singleton calls replaced with by_ref() captures in filter_map and map closures - sequence_payload: 1 cross_singleton replaced with by_ref() in map - index_payloads (sliced!): 1 cross_singleton replaced with by_ref() in map - acceptor_p2: 2 cross_singleton calls replaced with by_ref() captures in filter_map and map closures Left unchanged: - acceptor_p1: 2 cross_singleton calls remain because the a_log parameter comes from a forward_ref cycle, and by_ref() on cycle sources causes a graph partitioning error ("Handoff succ not in subgraph") Also added + 'a lifetime bounds where required by by_ref() usage: - PaxosLike::build, PaxosLike::with_client - paxos_core, index_payloads - compartmentalized_paxos_core Updated IR snapshots to reflect the new singleton ref nodes. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

- hydro_lang/src/compile/ir/mod.rs: Add an assert before the `ident_stack.drain()` subtraction to catch logical bugs where `ident_stack` is shorter than `singleton_refs`, avoiding a potential underflow in release builds. - hydro_lang/src/singleton_ref.rs: Fix missing space (`=SINGLETON_REFS` → `= SINGLETON_REFS`) so rustfmt formatting check passes. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2929

Added `with_singleton_capture` wrapper to `Stream::filter_map` so that `SingletonRef` values can be captured inside `filter_map` closures via `q!()`, matching the existing support in `map`, `filter`, `flat_map_ordered`, `flat_map_unordered`, `partition`, and `inspect`. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

…eton mechanism [ci-bench] We need to remove the global singleton counter and instead use a local counter. Converted 6 out of 7 cross_singleton usages in paxos.rs to use the new by_ref() singleton mechanism: - recommit_after_leader_election: 3 cross_singleton calls replaced with by_ref() captures in filter_map and map closures - sequence_payload: 1 cross_singleton replaced with by_ref() in map - index_payloads (sliced!): 1 cross_singleton replaced with by_ref() in map - acceptor_p2: 2 cross_singleton calls replaced with by_ref() captures in filter_map and map closures Left unchanged: - acceptor_p1: 2 cross_singleton calls remain because the a_log parameter comes from a forward_ref cycle, and by_ref() on cycle sources causes a graph partitioning error ("Handoff succ not in subgraph") Also added + 'a lifetime bounds where required by by_ref() usage: - PaxosLike::build, PaxosLike::with_client - paxos_core, index_payloads - compartmentalized_paxos_core Updated IR snapshots to reflect the new singleton ref nodes. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2903

MingweiSamuel force-pushed the mingwei/paxos-ref branch from 9a6bb50 to a6d6079 Compare May 29, 2026 19:43

MingweiSamuel changed the base branch from main to mingwei/remote-old-singleton-all June 3, 2026 22:09

Base automatically changed from mingwei/remote-old-singleton-all to main June 3, 2026 22:40

MingweiSamuel force-pushed the mingwei/paxos-ref branch from a6d6079 to 078d557 Compare June 5, 2026 20:25

MingweiSamuel changed the base branch from main to mingwei/singletonref-counter June 5, 2026 20:25

MingweiSamuel force-pushed the mingwei/singletonref-counter branch from e530702 to b0db6d5 Compare June 5, 2026 20:25

MingweiSamuel force-pushed the mingwei/singletonref-counter branch from b0db6d5 to 84b2c09 Compare June 5, 2026 22:07

MingweiSamuel force-pushed the mingwei/paxos-ref branch from 078d557 to adac487 Compare June 5, 2026 22:07

MingweiSamuel force-pushed the mingwei/singletonref-counter branch from 84b2c09 to 74e5643 Compare June 8, 2026 17:18

MingweiSamuel force-pushed the mingwei/singletonref-counter branch from 74e5643 to 123a9c6 Compare June 8, 2026 17:54

MingweiSamuel force-pushed the mingwei/paxos-ref branch from adac487 to 23ea024 Compare June 8, 2026 17:54

MingweiSamuel and others added 3 commits June 10, 2026 19:31

MingweiSamuel force-pushed the mingwei/paxos-ref branch from 23ea024 to 2f9004d Compare June 10, 2026 19:31

Base automatically changed from mingwei/singletonref-counter to main June 10, 2026 20:21

MingweiSamuel mentioned this pull request Jun 11, 2026

feat(hydro_lang): add by_ref() singleton capture support to Stream::filter_map #2937

Closed

MingweiSamuel changed the base branch from main to mingwei/filtermap-ref June 11, 2026 16:31

MingweiSamuel changed the base branch from mingwei/filtermap-ref to main June 12, 2026 16:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(hydro_test): migrate paxos cross_singleton to by_ref() singleton mechanism [ci-bench]#2903

refactor(hydro_test): migrate paxos cross_singleton to by_ref() singleton mechanism [ci-bench]#2903
MingweiSamuel wants to merge 4 commits into
mainfrom
mingwei/paxos-ref

MingweiSamuel commented May 28, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented May 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 28, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

MingweiSamuel commented May 28, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying hydro with Cloudflare Pages

Uh oh!

github-actions Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Benchmark Results

Run History:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cloudflare-workers-and-pages Bot commented May 28, 2026 •

edited

Loading

github-actions Bot commented May 28, 2026 •

edited

Loading