Skip to content

feat(hydro_lang): multi-versioning support in the simulator#2945

Open
luckyworkama wants to merge 1 commit into
mainfrom
push-mxwwtmrsnmzm
Open

feat(hydro_lang): multi-versioning support in the simulator#2945
luckyworkama wants to merge 1 commit into
mainfrom
push-mxwwtmrsnmzm

Conversation

@luckyworkama

@luckyworkama luckyworkama commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@luckyworkama luckyworkama requested a review from a team June 17, 2026 04:52
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown

Deploying hydro with  Cloudflare Pages  Cloudflare Pages

Latest commit: 2ebc4d6
Status: ✅  Deploy successful!
Preview URL: https://f290b647.hydroflow.pages.dev
Branch Preview URL: https://push-mxwwtmrsnmzm.hydroflow.pages.dev

View logs

@luckyworkama luckyworkama marked this pull request as draft June 17, 2026 04:52
@luckyworkama luckyworkama force-pushed the push-mxwwtmrsnmzm branch 7 times, most recently from 8f60b45 to 653854f Compare June 17, 2026 23:38
@luckyworkama luckyworkama marked this pull request as ready for review June 17, 2026 23:49
@luckyworkama luckyworkama force-pushed the push-mxwwtmrsnmzm branch 2 times, most recently from cafb97a to 14e48b2 Compare June 17, 2026 23:55
shadaj
shadaj reviewed Jun 18, 2026
View reviewed changes

@shadaj shadaj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass, have not reviewed the send half and recv half bits yet.

Comment thread hydro_test/src/distributed/versioning.rs Outdated
Comment thread hydro_test/src/distributed/versioning.rs
Comment thread hydro_test/src/distributed/versioning.rs Outdated
Comment thread hydro_lang/src/sim/graph.rs
Comment thread hydro_lang/src/compile/builder.rs Outdated
Comment thread hydro_lang/src/compile/ir/mod.rs Outdated
Comment thread hydro_lang/src/compile/ir/mod.rs Outdated
@luckyworkama

luckyworkama commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

hmm, deriving the version from an existing cluster seems a bit weird. In the sim_multi_version_differing_topology test, it creates 3 versions, gossip_v1, gossip_v2 and gossip_v3 (which includes a new Logger cluster). The issue is that the Logger cluster is internally going to have a v0 stamp on it, while gossip_v3 will have a v3 stamp on it. This wasn't the case with the flow-wide versioning api before.

I think it is not actually an issue? It does feel kinda weird tho. @shadaj

luckyworkama added a commit that referenced this pull request Jun 18, 2026
@luckyworkama @infinity-hydro
feat(hydro_lang): multi-versioning support in the simulator
b57af73 
Co-authored-by: Infinity <infinity@hydro.run>

Address shadaj's review comments on PR #2945 (multi-versioning sim)

hydro_lang/src/compile/ir/mod.rs:
- Change the two `panic!`s in the non-simulation builder's
  `create_versioned_network_fork` / `create_versioned_network` stubs to
  `unreachable!`, since these nodes are only produced by the simulator merge pass.

hydro_lang/src/compile/builder.rs:
- Fix `next_version` to compute the next version number from the highest version
  already assigned to any location in the correspondence group, rather than from
  the `from` cluster directly. This avoids version-number collisions when
  `next_version` is called more than once on the same source cluster.

hydro_test/src/distributed/versioning.rs:
- Replace the three near-identical `gossip_server_v1/v2/v3` functions with a single
  `gossip_server_versioned` that takes a `GossipVersion` enum and branches internally
  (matching the approach real versioned code would take), updating all call sites.
- Remove the redundant `sim_multi_version_basic_echo_fuzz` test, since Bolero
  guarantees fuzz explores a subset of exhaustive.
- Add `sim_multi_version_two_clusters_differing_version_counts`, exercising two
  independent clusters with different numbers of versions (2 and 3) to guard against
  bugs when version counts differ across clusters.

Co-authored-by: Infinity 🤖 <infinity@hydro.run>
luckyworkama added a commit that referenced this pull request Jun 18, 2026
@luckyworkama @infinity-hydro
feat(hydro_lang): multi-versioning support in the simulator
2c82a18 
Co-authored-by: Infinity <infinity@hydro.run>

Address shadaj's review comments on PR #2945 (multi-versioning sim)

hydro_lang/src/compile/ir/mod.rs:
- Change the two `panic!`s in the non-simulation builder's
  `create_versioned_network_fork` / `create_versioned_network` stubs to
  `unreachable!`, since these nodes are only produced by the simulator merge pass.

hydro_lang/src/compile/builder.rs:
- Fix `next_version` to compute the next version number from the highest version
  already assigned to any location in the correspondence group, rather than from
  the `from` cluster directly. This avoids version-number collisions when
  `next_version` is called more than once on the same source cluster.

hydro_test/src/distributed/versioning.rs:
- Replace the three near-identical `gossip_server_v1/v2/v3` functions with a single
  `gossip_server_versioned` that takes a `GossipVersion` enum and branches internally
  (matching the approach real versioned code would take), updating all call sites.
- Remove the redundant `sim_multi_version_basic_echo_fuzz` test, since Bolero
  guarantees fuzz explores a subset of exhaustive.
- Add `sim_multi_version_two_clusters_differing_version_counts`, exercising two
  independent clusters with different numbers of versions (2 and 3) to guard against
  bugs when version counts differ across clusters.

Co-authored-by: Infinity 🤖 <infinity@hydro.run>
@shadaj

shadaj commented Jun 18, 2026

Copy link
Copy Markdown
Member

hmm, deriving the version from an existing cluster seems a bit weird. In the sim_multi_version_differing_topology test, it creates 3 versions, gossip_v1, gossip_v2 and gossip_v3 (which includes a new Logger cluster). The issue is that the Logger cluster is internally going to have a v0 stamp on it, while gossip_v3 will have a v3 stamp on it. This wasn't the case with the flow-wide versioning api before.

I think it is not actually an issue? It does feel kinda weird tho. @shadaj

Honestly, I think it's quite elegant! Basically each cluster can have a different number / set of versions, it's akin to having one dynamic versioning flag per-cluster.

luckyworkama added a commit that referenced this pull request Jun 22, 2026
@luckyworkama @infinity-hydro
feat(hydro_lang): multi-versioning support in the simulator
b450b9e 
Co-authored-by: Infinity <infinity@hydro.run>

Address shadaj's review comments on PR #2945 (multi-versioning sim)

hydro_lang/src/compile/ir/mod.rs:
- Change the two `panic!`s in the non-simulation builder's
  `create_versioned_network_fork` / `create_versioned_network` stubs to
  `unreachable!`, since these nodes are only produced by the simulator merge pass.

hydro_lang/src/compile/builder.rs:
- Fix `next_version` to compute the next version number from the highest version
  already assigned to any location in the correspondence group, rather than from
  the `from` cluster directly. This avoids version-number collisions when
  `next_version` is called more than once on the same source cluster.

hydro_test/src/distributed/versioning.rs:
- Replace the three near-identical `gossip_server_v1/v2/v3` functions with a single
  `gossip_server_versioned` that takes a `GossipVersion` enum and branches internally
  (matching the approach real versioned code would take), updating all call sites.
- Remove the redundant `sim_multi_version_basic_echo_fuzz` test, since Bolero
  guarantees fuzz explores a subset of exhaustive.
- Add `sim_multi_version_two_clusters_differing_version_counts`, exercising two
  independent clusters with different numbers of versions (2 and 3) to guard against
  bugs when version counts differ across clusters.

Co-authored-by: Infinity 🤖 <infinity@hydro.run>
@luckyworkama @infinity-hydro
feat(hydro_lang): multi-versioning support in the simulator
2ebc4d6 
Co-authored-by: Infinity <infinity@hydro.run>

Address shadaj's review comments on PR #2945 (multi-versioning sim)

hydro_lang/src/compile/ir/mod.rs:
- Change the two `panic!`s in the non-simulation builder's
  `create_versioned_network_fork` / `create_versioned_network` stubs to
  `unreachable!`, since these nodes are only produced by the simulator merge pass.

hydro_lang/src/compile/builder.rs:
- Fix `next_version` to compute the next version number from the highest version
  already assigned to any location in the correspondence group, rather than from
  the `from` cluster directly. This avoids version-number collisions when
  `next_version` is called more than once on the same source cluster.

hydro_test/src/distributed/versioning.rs:
- Replace the three near-identical `gossip_server_v1/v2/v3` functions with a single
  `gossip_server_versioned` that takes a `GossipVersion` enum and branches internally
  (matching the approach real versioned code would take), updating all call sites.
- Remove the redundant `sim_multi_version_basic_echo_fuzz` test, since Bolero
  guarantees fuzz explores a subset of exhaustive.
- Add `sim_multi_version_two_clusters_differing_version_counts`, exercising two
  independent clusters with different numbers of versions (2 and 3) to guard against
  bugs when version counts differ across clusters.

Co-authored-by: Infinity 🤖 <infinity@hydro.run>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shadaj shadaj shadaj left review comments

At least 0 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@luckyworkama @shadaj