feat(hydro_lang): add forward consistency analysis for IR sinks#2927
Open
jhellerstein wants to merge 1 commit into
Open
feat(hydro_lang): add forward consistency analysis for IR sinks#2927jhellerstein wants to merge 1 commit into
jhellerstein wants to merge 1 commit into
Conversation
jhellerstein
force-pushed
the
feat/drop-consistency-stream
branch
from
12a6f18 to
49303cb
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
from
5d04b46 to
68deb32
Compare
Deploying hydro with
|
| Latest commit: |
6600441
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://738f83bc.hydroflow.pages.dev |
| Branch Preview URL: | https://feat-consistency-label-analy.hydroflow.pages.dev |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an in-memory IR consistency labeling API for Hydro sinks, aiming to match the external coord-analysis labeling while also providing blame spans for unresolved nondeterminism.
Changes:
- Introduces
compile::consistency_labelwith label derivation, sink label extraction, and nondeterminism scanning. - Exposes new
BuiltFlowAPIs for (a) type-metadata-based sink labels and (b) forward (sink-oriented) analysis that downgrades labels on unresolved untrusted nondeterminism. - Adds unit + integration + simulator tests to validate label derivation and (partially) alignment with runtime behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| hydro_lang/src/compile/mod.rs | Exposes the new consistency_label module from compile. |
| hydro_lang/src/compile/consistency_label.rs | Implements label derivation + extraction/analysis + tests. |
| hydro_lang/src/compile/built.rs | Adds public BuiltFlow::{sink_consistency_labels, analyze_consistency} APIs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jhellerstein
force-pushed
the
feat/drop-consistency-stream
branch
from
49303cb to
ed4d130
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
from
68deb32 to
2707eb4
Compare
jhellerstein
force-pushed
the
feat/drop-consistency-stream
branch
from
ed4d130 to
f713343
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
2 times, most recently
from
a1eddc2 to
d552a26
Compare
jhellerstein
changed the base branch from
feat/drop-consistency-stream
to
feat/is-proved-fold-properties
jhellerstein
force-pushed
the
feat/is-proved-fold-properties
branch
from
f26d93a to
feaf0f8
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
from
d552a26 to
81bc818
Compare
jhellerstein
force-pushed
the
feat/is-proved-fold-properties
branch
from
feaf0f8 to
1d58e7b
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
2 times, most recently
from
f1a7765 to
78ce814
Compare
jhellerstein
force-pushed
the
feat/is-proved-fold-properties
branch
from
1d58e7b to
cc07401
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
2 times, most recently
from
3526f92 to
5c49f5b
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
2 times, most recently
from
15e0816 to
5863b64
Compare
jhellerstein
force-pushed
the
feat/is-proved-fold-properties
branch
from
cc07401 to
71bbf8b
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
3 times, most recently
from
8ff063d to
65b6a1b
Compare
jhellerstein
force-pushed
the
feat/is-proved-fold-properties
branch
from
2dd09ab to
fb5d5ae
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
from
65b6a1b to
9abf61a
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
from
9abf61a to
d434d8e
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
from
d434d8e to
0b55571
Compare
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
from
0e2a973 to
f707981
Compare
Adds a forward consistency analysis that operates directly on the in-memory IR, producing labels equivalent to the external coord-analysis tool. For each observable sink, derives a label from the output type and walks backward checking for untrusted ObserveNonDet nodes not resolved by a c+i fold. Infers commutativity/idempotency from the fold's input collection_kind (NoOrder → commutative, AtLeastOnce → idempotent) per discussion with @shadaj. Also adds FlowBuilder::cluster_with_consistency() for creating clusters with explicit consistency types.
jhellerstein
force-pushed
the
feat/consistency-label-analysis
branch
from
b9110c2 to
6600441
Compare
jhellerstein
commented
Jun 8, 2026
jhellerstein
left a comment
jhellerstein
left a comment
Contributor
Author
There was a problem hiding this comment.
redundancy in API: extract_sink_labels was the first version (metadata-based, for comparing with coord-analysis JSON output). analyze_sink_labels supersedes it — it does the actual analysis AND includes all sinks.
we should just remove extract_sink_labels and sink_consistency_labels, keeping only analyze_sink_labels / analyze_consistency.
MingweiSamuel
force-pushed
the
main
branch
2 times, most recently
from
a15a670 to
e70eab6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a forward consistency analysis that operates directly on the in-memory IR, producing labels equivalent to the external
coord-analysistool.For each observable sink, the analysis derives a consistency label from the output type (ordering + retry) and walks backward checking for untrusted
ObserveNonDetnodes not resolved by a commutative+idempotent fold. Commutativity/idempotency are inferred from the fold's inputCollectionKind(NoOrder→ commutative,AtLeastOnce→ idempotent), avoiding extra fields on IR nodes.Labels:
SEQ_CONSISTENT,PER_KEY_SEQ_CONSISTENT,CONVERGENT_MULTISET,CONVERGENT_SET,CONVERGENT_LATTICE,INCONSISTENT.Also adds
FlowBuilder::cluster_with_consistency()for creating clusters with explicit consistency types.Includes 8 tests (unit, integration with
broadcast_closed, and simulator-validated).