iterators: add lazy tree traversals Iterators.dfs and Iterators.bfs - #20
Open
adienes wants to merge 3 commits into
Open
iterators: add lazy tree traversals Iterators.dfs and Iterators.bfs#20adienes wants to merge 3 commits into
Iterators.dfs and Iterators.bfs#20adienes wants to merge 3 commits into
Conversation
adienes
force-pushed
the
iterators-dfs-bfs
branch
9 times, most recently
from
July 26, 2026 15:19
3da8f01 to
03ab5d7
Compare
Add lazy depth-first (preorder, postorder, or leaves-only) and breadth-first iterators over trees whose structure is given by a `children` function, with an optional `visited` set that extends them to DAGs and cyclic graphs. The traversals are fully lazy: child collections are advanced one element at a time and a node's children are not requested before the node itself has been yielded, so unbounded trees and expensive `children` functions (e.g. filesystem access) are handled gracefully. The iterate protocol must carry every level of the descent in one state value, so the state is a stack (or queue) of dynamically typed frames holding each level's live child iterator, and each step pays dynamic dispatch; whole-traversal consumption at recursion speed is provided separately by the internal-iteration driver in the following commits. Use the new iterators to simplify hand-rolled traversals throughout Base and the stdlibs, with no behavior changes: - `walkdir` still returns the same `Channel`, but its producer is now a `dfs` over `(path, dirs, files)` triples instead of a hand-written recursion, with `topdown` mapping onto the traversal order. - `Base.Precompilation` had four near-duplicate hand-rolled dependency-closure walks (`collect_all_deps`, `visit_indirect_deps!`, `_collect_reachable!`, and the reverse-dependency frontier loop in the precompile report); they are now expressed directly as `dfs`/`bfs` traversals sharing visited sets. - `Profile`'s `StackFrameTree` cleanup and max-statistics sweeps use `bfs` and `Test`'s `filter_errors` testset walk uses `dfs` instead of explicit stacks and recursion. This commit was written with the assistance of generative AI (Claude). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add `Iterators._iterate_loop(body, itr, accum)`: a loop driver in which `body` returns a control token (`LoopContinue`/`LoopBreak`/`LoopReturn`) and loop-carried variables thread through `accum`. The default method reproduces the external-protocol `for` loop; `DepthFirst`/`BreadthFirst` overload it with fused recursive traversal, which keeps every frame's iterator and state as typed locals on the call stack — something the external iterate protocol cannot express, because its state must carry all levels of the descent in one value. Internal API: the next commit routes the standard consumers through it. This commit was written with the assistance of generative AI (Claude). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
adienes
force-pushed
the
iterators-dfs-bfs
branch
from
July 26, 2026 20:29
03ab5d7 to
01cb2ad
Compare
Overload `_foldl_impl`, `foreach`, `grow_to!`, `_collect`, `_any`, and `_all` for the tree traversals so that `foreach`, `collect`, `map`, comprehensions, `Set`, and the `foldl`-based reductions (`sum`, `count`, `maximum`, `reduce`, `mapreduce`, `any`, `all`, `in`, ...) drive `Iterators.dfs` and `Iterators.bfs` through `_iterate_loop` instead of the iterate protocol. `mapfoldl` already strips `Generator`/`Filter`/ `Flatten` wrappers into reducing functions before folding the innermost iterator, which covers compositions such as `sum(f, Iterators.filter(p, dfs(...)))`; the collectors get the same unwrapping, so `map` and comprehensions over a traversal fold it too, and dispatch back to the generic methods statically when the innermost iterator is not a traversal. (Typed `T[...]` comprehensions lower to inline protocol loops rather than a `collect` call and are unchanged; explicit `collect(T, itr)` takes the fast path.) These are the choke points whole consumer families route through with no alternative spelling; leaf consumers such as `union!` are left on the protocol path, where `foreach` with `push!` is the one-line fast alternative. On a heterogeneous tree the consumers run an order of magnitude faster than element-by-element iteration (foreach 14ns/node vs 264, count 17, collect 23), within 4x of hand-fused recursion; short-circuiting consumers exit early through the driver control tokens. This is the same design as the `mapreduce` specialization for `skipmissing`: the iterate protocol stays the universal composable path, and consumers that can push instead of pull dispatch to the fused recursion. This commit was written with the assistance of generative AI (Claude). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
adienes
force-pushed
the
iterators-dfs-bfs
branch
from
July 27, 2026 13:10
01cb2ad to
e84808c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add lazy depth-first (preorder or postorder) and breadth-first iterators over trees whose structure is given by a
childrenfunction, with an optionalvisitedset that extends them to DAGs and cyclic graphs. The traversals are fully lazy: child collections are advanced one element at a time and a node's children are not requested before the node itself has been yielded, so unbounded trees and expensivechildrenfunctions (e.g. filesystem access) are handled gracefully.Use them to replace hand-rolled traversals throughout Base and the stdlibs:
walkdiris now a thin stateful wrapper (preserving the documented resume-on-repeated-access behavior as well astake!) around adfsover(path, dirs, files)triples, instead of recursing inside a task-backedChannel. No filesystem access happens before the first iteration, andonerrorfailures surface at the exact iteration that encountered them rather than through the channel machinery.Base.Precompilationhad four near-duplicate hand-rolled dependency-closure walks (collect_all_deps,visit_indirect_deps!,_collect_reachable!, and the reverse-dependency frontier loop in the precompile report); they are now expressed directly asdfs/bfstraversals sharing visited sets.Profile'sStackFrameTreecleanup and max-statistics sweeps andTest'sfilter_errorstestset walk usedfsinstead of explicit stacks and recursion.This commit was written with the assistance of generative AI (Claude).