refactor(dfir_lang): remove stratum, add push codegen, test fold, fold_keyed, fold_no_replay#2965
Open
MingweiSamuel wants to merge 4 commits into
Open
refactor(dfir_lang): remove stratum, add push codegen, test fold, fold_keyed, fold_no_replay#2965MingweiSamuel wants to merge 4 commits into
fold, fold_keyed, fold_no_replay#2965MingweiSamuel wants to merge 4 commits into
Conversation
Deploying hydro with
|
| Latest commit: |
d0bf7c3
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://115b91c2.hydroflow.pages.dev |
| Branch Preview URL: | https://mingwei-newpush-fold.hydroflow.pages.dev |
MingweiSamuel
force-pushed
the
mingwei/newpush-fold
branch
from
909fde1 to
ecf3876
Compare
MingweiSamuel
force-pushed
the
mingwei/newpush-fold
branch
2 times, most recently
from
f810461 to
571efde
Compare
Member
Author
|
Diff view: 571efde |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors DFIR language/codegen and pipes to support push-side codegen for blocking/stateful operators (notably fold, fold_keyed, and fold_no_replay), removing the prior stratum-based scheduling requirement and adding new push-side combinators plus tests to validate behavior.
Changes:
- Add unified push-side accumulator infrastructure (
Accumulate+AccumState) and new push combinators (Sort,FoldKeyed,ReduceKeyed), then wire DFIR codegen to use them for push-sidefold*operators. - Update/extend tests to cover push-side blocking behavior and adjust existing metrics expectations.
- Minor utility/API improvements (extra derives) and CI/precheck target updates.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| precheck.bash | Adds hydro_test_embedded to the hydro test targets run by precheck. |
| dfir_rs/tests/surface_push_blocking.rs | New tests ensuring fold, fold_keyed, and fold_no_replay behave correctly when they end up on the push side of a subgraph. |
| dfir_rs/tests/metrics.rs | Updates the handoff-metrics test pipeline and expected output. |
| dfir_rs/tests/compile-fail/stable/surface_fold_keyed_generics_bad.stderr | Updates expected compiler stderr output for trybuild tests. |
| dfir_rs/tests/compile-fail/stable/surface_fold_keyed_badtype_option.stderr | Updates expected compiler stderr output for trybuild tests. |
| dfir_rs/tests/compile-fail/stable/surface_fold_keyed_badtype_int.stderr | Updates expected compiler stderr output for trybuild tests. |
| dfir_rs/tests/compile-fail/stable/surface_demuxenum_wrongfields_2.stderr | Updates expected compiler stderr output for trybuild tests. |
| dfir_rs/tests/compile-fail/stable/surface_demuxenum_wrongfields_1.stderr | Updates expected compiler stderr output for trybuild tests. |
| dfir_rs/tests/compile-fail/stable/surface_demuxenum_wrongenum.stderr | Updates expected compiler stderr output for trybuild tests. |
| dfir_rs/src/util/sparse_vec.rs | Adds Clone/Debug derives for SparseVec. |
| dfir_rs/src/util/mod.rs | Adds common derives for Persistence and PersistenceKeyed. |
| dfir_pipes/src/push/sort.rs | Introduces a push-side Sort combinator with backpressure-aware finalize draining. |
| dfir_pipes/src/push/reduce_keyed.rs | Introduces a push-side ReduceKeyed combinator that flushes reduced (K,V) pairs on finalize. |
| dfir_pipes/src/push/mod.rs | Registers new push modules/re-exports and adds constructors for fold/reduce/accumulate/sort. |
| dfir_pipes/src/push/fold_keyed.rs | Introduces a push-side FoldKeyed combinator that flushes (K, Acc) pairs on finalize. |
| dfir_pipes/src/push/accumulate.rs | Adds the unified Accumulate push combinator and AccumState trait. |
| dfir_pipes/src/push/accum_state.rs | Adds AccumState implementations for fold/reduce/sort patterns. |
| dfir_lang/src/graph/ops/fold.rs | Removes stratum delay and adds push-side codegen path using the new push accumulator infrastructure. |
| dfir_lang/src/graph/ops/fold_no_replay.rs | Removes stratum delay and adds push-side codegen path for no-replay semantics. |
| dfir_lang/src/graph/ops/fold_keyed.rs | Removes stratum delay and adds push-side codegen path using the new FoldKeyed push combinator. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+156
to
+166
| #root::dfir_pipes::push::filter( | ||
| { | ||
| let __was_updated = &#was_updated_ident; | ||
| let __context: &_ = #context; | ||
| move |_| __was_updated.get() || __context.current_tick().0 == 0 | ||
| }, | ||
| #root::dfir_pipes::push::map( | ||
| |__val: &mut _| ::std::clone::Clone::clone(&*__val), | ||
| #output, | ||
| ), | ||
| ), |
Comment on lines
+136
to
+145
| // Drain the iterator into downstream, respecting backpressure. | ||
| if let AccumPhase::Draining(iter) = this.phase { | ||
| loop { | ||
| ready!(this.next.as_mut().poll_ready(ctx)); | ||
| let Some(item) = iter.next() else { | ||
| break; | ||
| }; | ||
| this.next.as_mut().start_send(item, ()); | ||
| } | ||
| *this.phase = AccumPhase::Done; |
MingweiSamuel
force-pushed
the
mingwei/newpush-fold
branch
from
571efde to
36df39f
Compare
…ialOrd)]` for `Persistence`, `PersistenceKeyed`, and `SparseVec` PR: #2964
MingweiSamuel
force-pushed
the
mingwei/newpush-fold
branch
from
36df39f to
18965f1
Compare
…fold_keyed`, `fold_no_replay` all push ops PR: #2965
MingweiSamuel
force-pushed
the
mingwei/newpush-fold
branch
from
18965f1 to
d0bf7c3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.