Feature/dynamic - #68
Open
jaredraycoleman wants to merge 13 commits into
Open
Conversation
… fastest node, fcp, heft, maxmin, minmin
… an example for delaying a task indefinitely
…preemptive)
Consolidate the dynamic-workflow schedulers into three families that reschedule
when a new task graph arrives, each with a different policy:
- preemptive (p*): recompose all arrived graphs and fully reschedule every task
that has not yet started.
- npreemptive (np*): additive/residual — place each new graph's tasks on top of
the existing schedule; never move an already-placed task.
(Replaces the former residual/ package.)
- kpreemptive (kp*): last-k preemptive — reschedule only within a window of the
last k arrived graphs (k is a constructor argument).
Each family provides base-heuristic variants (heft, cpop, minmin, maxmin, flb,
random, ...). Self-driving variants subclass DWScheduler and take the full
List[(task_graph, arrival_time)]; incremental variants subclass Scheduler and are
driven one graph at a time by a residual wrapper.
Verified: all modules import cleanly; the self-driving DWScheduler variants
produce valid schedules (dependencies respected, no per-node overlap, arrival
times honored) across multiple random instances, with makespan ordering
preemptive <= kpreemptive <= npreemptive as expected.
Known issue: preemptive/PFastestNodeScheduler double-places a task across
rescheduling rounds (left as-is for follow-up).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces infrastructure and new scheduler implementations aimed at supporting dynamic task-graph arrivals (including preemptive / non-preemptive / k-window variants), plus adds utilities for CCR-based network-weight scaling and expands the parametric scheduler API to support residual/cumulative scheduling via a min_start_time offset.
Changes:
- Add
add_ccr_weights(...)to scale network edge weights to a target communication-to-computation ratio (CCR). - Extend
ParametricScheduler.schedule(...)withmin_start_timeand acumulativemode, and adjust insertion logic to better handle negative/offset times. - Add many new dynamic task-graph schedulers under
src/saga/schedulers/dynamic_task_graph/**and new example scripts underscripts/examples/**.
Reviewed changes
Copilot reviewed 45 out of 134 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/saga/utils/random_graphs.py | Adds add_ccr_weights helper to scale network link weights based on CCR. |
| src/saga/schedulers/wba.py | Ensures returned schedule includes empty lists for all network nodes. |
| src/saga/schedulers/parametric/components.py | Adjusts append-only start-time calculation to clamp negative end times. |
| src/saga/schedulers/parametric/init.py | Adds cumulative + min_start_time support and schedule time shifting. |
| src/saga/schedulers/flb.py | Minor docstring wording fix. |
| src/saga/schedulers/dynamic_task_graph/preemptive/prandom.py | Adds preemptive random dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/preemptive/pminmin.py | Adds preemptive MinMin dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/preemptive/pmaxmin.py | Adds preemptive MaxMin dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/preemptive/pheft.py | Adds preemptive HEFT-like dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/preemptive/pflb.py | Adds preemptive FLB-like dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/preemptive/pfastest_node.py | Adds preemptive fastest-node dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/preemptive/pcpop.py | Adds preemptive CPOP-like dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npwba.py | Adds non-preemptive residual WBA dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npsufferage.py | Adds non-preemptive residual Sufferage dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npsmt.py | Adds SMT-based scheduler (currently not aligned with dynamic interface). |
| src/saga/schedulers/dynamic_task_graph/npreemptive/nprandom.py | Adds non-preemptive random dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npolb.py | Adds residual OLB dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npmsbc.py | Adds residual MSBC dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npminmin.py | Adds non-preemptive MinMin dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npmet.py | Adds residual MET dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npmct.py | Adds residual MCT dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npmaxmin.py | Adds non-preemptive MaxMin dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/nphybrid.py | Adds placeholder hybrid scheduler (currently pass). |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npheft.py | Adds non-preemptive HEFT-like dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/nphbmct.py | Adds residual HBMCT dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npgdl.py | Adds residual GDL dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npflb.py | Adds non-preemptive FLB-like dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npfastest_node.py | Adds non-preemptive fastest-node dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npetf.py | Adds residual ETF dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npduplex.py | Adds residual duplex dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npdps.py | Adds residual DPS dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npfcp.py | Adds residual FCP dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npcpop.py | Adds non-preemptive CPOP-like dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npbrute_force.py | Adds residual brute-force dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/npreemptive/npbil.py | Adds residual BIL dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/kpreemptive/kprandom.py | Adds k-window preemptive random dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/kpreemptive/kpminmin.py | Adds k-window preemptive MinMin dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/kpreemptive/kpmaxmin.py | Adds k-window preemptive MaxMin dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/kpreemptive/kpheft.py | Adds k-window preemptive HEFT-like dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/kpreemptive/kpflb.py | Adds k-window preemptive FLB-like dynamic scheduler. |
| src/saga/schedulers/dynamic_task_graph/kpreemptive/kpcpop.py | Adds k-window preemptive CPOP-like dynamic scheduler. |
| src/saga/scheduler.py | Adds ResidualScheduler base class (but missing DWScheduler used elsewhere). |
| scripts/examples/spare_core/main.py | Adds large example comparing schedulers / “spare core” experimentation. |
| scripts/examples/parametric_residual/main.py | Adds parametric residual vs cumulative example using min_start_time + CCR weights. |
| scripts/examples/dynamic_task_graph/main.py | Adds dynamic task-graph example script (imports appear out of date). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+40
to
42
|
|
||
|
|
||
|
|
Comment on lines
+128
to
130
|
|
||
|
|
||
| queue = self.initial_priority(network, task_graph) |
Comment on lines
+153
to
+155
| if task.start < 0: | ||
| queue.remove(task.name) | ||
|
|
Comment on lines
+27
to
+41
| network = deepcopy(network) | ||
| mean_network_weight = np.mean([ | ||
| network.nodes[node]["weight"] | ||
| for node in network.nodes | ||
| ]) | ||
| mean_task_cost = np.mean([ | ||
| task_graph.nodes[node]["weight"] | ||
| for node in task_graph.nodes | ||
| ]) | ||
| mean_dependency_weight = np.mean([ | ||
| task_graph.edges[edge]["weight"] | ||
| for edge in task_graph.edges | ||
| ]) | ||
|
|
||
| link_strength = (mean_dependency_weight * mean_network_weight) / (ccr * mean_task_cost) |
Comment on lines
+6
to
+10
| from saga.scheduler import Task | ||
|
|
||
| from ....scheduler import Task | ||
| from ....scheduler import Scheduler | ||
|
|
Comment on lines
+10
to
+13
| from saga.scheduler import Task | ||
|
|
||
| from ....scheduler import Scheduler, Task, DWScheduler | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type of change
Checklist
mainand up to date with it (no conflicts)uv run pytest tests/ --timeout=120passes locallyuv run mypy src/saga --ignore-missing-importsis cleanuv run ruff check src/sagaanduv run ruff format --check src/sagaare cleanpyproject.toml/uv.lockchanges, or version bumpNotes for reviewers