Skip to content

backend,nix,frontend: feat: build resource usage from the Nix daemon and worker transfer speeds #714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DerDennisOP merged 22 commits into from
Oct 4, 2026
Merged

backend,nix,frontend: feat: build resource usage from the Nix daemon and worker transfer speeds #714

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
bfd82b9
backend,frontend,docs: fix: keep substituted outputs out of the build…
DerDennisOP Oct 4, 2026
156e596
backend,frontend,docs: feat: replace the per-build network peak with …
DerDennisOP Oct 4, 2026
ccfbc24
backend,nix,docs: feat: take build resource usage from the Nix daemon…
DerDennisOP Oct 4, 2026
f7022bd
nix: feat: report build resource usage and keep build statistics in t…
DerDennisOP Oct 4, 2026
3d37d87
docs: docs: add teams to the roadmap
DerDennisOP Oct 4, 2026
04b45e0
backend: fix: allow the harmonia fork as a git source
DerDennisOP Oct 4, 2026
5e95b96
backend: feat: report live build stages and the upload grant wait, an…
DerDennisOP Oct 4, 2026
6771ac6
backend: feat: learn storage throughput and the stored-to-NAR size ra…
DerDennisOP Oct 4, 2026
614697b
backend,nix: feat: limit NAR downloads from storage to 16 across all …
DerDennisOP Oct 4, 2026
e910445
backend: fix: fill the build resource usage fields in the mock daemon
DerDennisOP Oct 4, 2026
cf83f7c
backend: feat: rank builds by their estimated download, build and upl…
DerDennisOP Oct 4, 2026
5423046
docs,frontend: docs: describe the estimated time rule, build stages, …
DerDennisOP Oct 4, 2026
7a0b46c
backend: feat: replace the missing paths rule with a learned per-path…
DerDennisOP Oct 4, 2026
34838a4
backend: feat: fold realised outputs, out-of-memory risk, local build…
DerDennisOP Oct 4, 2026
6f84847
backend: feat: estimate evaluations by their recorded run time so bui…
DerDennisOP Oct 4, 2026
a6e1c36
backend: feat: hold builds with large transfers below the assignment …
DerDennisOP Oct 4, 2026
ae53d74
docs: docs: describe the per-path time, evaluation estimates, transfe…
DerDennisOP Oct 4, 2026
832ec8b
docs: docs: add gradient.ci servers to the roadmap and swap the v2.1 …
DerDennisOP Oct 4, 2026
25a4ca5
frontend: feat: spin the building status icon 2.5x as fast for priori…
DerDennisOP Oct 4, 2026
fdded01
backend: feat: report builds and evaluations as prioritized whenever …
DerDennisOP Oct 4, 2026
33cb6b7
gradient-wire,gradient-proto: feat: add handover and path list messag…
DerDennisOP Oct 4, 2026
86ec328
gradient-worker: feat: hide store paths off the current user's list, …
DerDennisOP Oct 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 25 additions & 34 deletions backend/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 14 additions & 14 deletions backend/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,20 +157,20 @@ russh = { version = "0.63", default-features = false, features = ["aws-lc-
shell-words = { version = "1.1", default-features = false, features = ["std"] }
socket2 = { version = "0.6", default-features = false }

harmonia-daemon = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-file-nar = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-protocol = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-store-aterm = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-store-content-address = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-store-derivation = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-store-db = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-store-path = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-store-path-info = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-store-remote = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-utils-base-encoding = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-utils-hash = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-utils-io = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-utils-signature = { git = "https://github.com/nix-community/harmonia.git", default-features = false }
harmonia-daemon = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-file-nar = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-protocol = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-store-aterm = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-store-content-address = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-store-derivation = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-store-db = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-store-path = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-store-path-info = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-store-remote = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-utils-base-encoding = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-utils-hash = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-utils-io = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }
harmonia-utils-signature = { git = "https://github.com/DerDennisOP/harmonia.git", branch = "build-resource-usage", default-features = false }

# Nix bindings. The version tracks the FORK below, not upstream: the patch
# replaces the source, so only the requirement is resolved against the index.
Expand Down
2 changes: 1 addition & 1 deletion backend/deny.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ ignore = []
unknown-registry = "deny"
unknown-git = "deny"
allow-git = [
"https://github.com/nix-community/harmonia",
"https://github.com/DerDennisOP/harmonia",
"https://github.com/DerDennisOP/nix-bindings",
"https://github.com/launchbadge/sqlx",
]
1 change: 1 addition & 0 deletions backend/gradient-cache/src/cacher/test_support.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ pub(crate) fn test_server_state_with_log(
git_host: gradient_git_host::GitHostRegistry::with_builtin(),
github_app_install_url: Default::default(),
upstream_query: Arc::new(tokio::sync::Semaphore::new(32)),
nar_downloads: std::sync::Arc::new(tokio::sync::Semaphore::new(16)),
upload_admission: gradient_storage::admission::UploadAdmission::new(
gradient_storage::admission::Limits {
concurrency: 16,
Expand Down
2 changes: 2 additions & 0 deletions backend/gradient-core/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ pub async fn init_state(cli: Cli) -> Result<Arc<ServerState>, InitError> {
};

let upstream_query_concurrency = config.cache.upstream_query_concurrency;
let nar_downloads = config.nar.max_concurrent_downloads.max(1);
let upload_limits = gradient_storage::admission::Limits {
concurrency: config.upload.concurrency.max(1),
bytes: config.upload.bytes_budget.max(1),
Expand All @@ -216,6 +217,7 @@ pub async fn init_state(cli: Cli) -> Result<Arc<ServerState>, InitError> {
upstream_query: Arc::new(tokio::sync::Semaphore::new(
upstream_query_concurrency.max(1),
)),
nar_downloads: Arc::new(tokio::sync::Semaphore::new(nar_downloads)),
upload_admission: gradient_storage::admission::UploadAdmission::new(upload_limits),
git_host: gradient_git_host::GitHostRegistry::with_builtin(),
github_app_install_url: Default::default(),
Expand Down
1 change: 1 addition & 0 deletions backend/gradient-core/src/state_root.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ pub struct AppState {
pub nar_storage: NarStore,
pub http: reqwest::Client,
pub upstream_query: Arc<Semaphore>,
pub nar_downloads: Arc<Semaphore>,
pub upload_admission: Arc<gradient_storage::admission::UploadAdmission>,
pub git_host: GitHostRegistry,
pub github_app_install_url: Arc<tokio::sync::OnceCell<String>>,
Expand Down
8 changes: 8 additions & 0 deletions backend/gradient-daemon/src/mock/build.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,10 @@ fn success(built_outputs: BuiltOutputs, started: i64) -> BuildResult {
stop_time: now_secs(),
cpu_user: None,
cpu_system: None,
memory_peak: None,
io_read_bytes: None,
io_write_bytes: None,
oom_kills: None,
}
}

Expand All @@ -264,6 +268,10 @@ fn failure(status: FailureStatus, msg: &str) -> BuildResult {
stop_time: 0,
cpu_user: None,
cpu_system: None,
memory_peak: None,
io_read_bytes: None,
io_write_bytes: None,
oom_kills: None,
}
}

Expand Down
4 changes: 4 additions & 0 deletions backend/gradient-daemon/src/mock/conn.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,10 @@ fn already_valid(path: &DerivedPath) -> KeyedBuildResult {
stop_time: 0,
cpu_user: None,
cpu_system: None,
memory_peak: None,
io_read_bytes: None,
io_write_bytes: None,
oom_kills: None,
},
}
}
Expand Down
Loading
Loading