Skip to content

Fix srun failure in the killarney imagenet job script - #222

Open
lebrice wants to merge 1 commit into
masterfrom
fix-killarney-srun-mem
Open

lebrice wants to merge 1 commit into
masterfrom
fix-killarney-srun-mem

Conversation

@lebrice

@lebrice lebrice commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On Killarney, the imagenet example job dies at its first srun:

srun: fatal: SLURM_MEM_PER_CPU, SLURM_MEM_PER_GPU, and SLURM_MEM_PER_NODE are mutually exclusive.

Killarney's lua job submit plugin adds a default --mem (8G) even when --mem-per-gpu is passed. The job then has both SLURM_MEM_PER_NODE=8192 and SLURM_MEM_PER_GPU=65536 set, and every srun inside it inherits both.

Fix

Request --mem=64G (per node) instead of --mem-per-gpu=64G in scripts/job_killarney.sh. For the default 1-GPU job the amount of memory is the same.

Verification

Test jobs on Killarney with the same resource flags:

  • --mem-per-gpu=64G (job 5684573): the job gets SLURM_MEM_PER_GPU=65536 and SLURM_MEM_PER_NODE=8192, and srun true fails with the error above.
  • --mem=64G (job 5684592): the job gets only SLURM_MEM_PER_NODE=65536, and srun true succeeds.

🤖 Generated with Claude Code

Killarney's job submit plugin adds a default per-node memory even when
--mem-per-gpu is given, so both SLURM_MEM_PER_NODE and SLURM_MEM_PER_GPU
end up set in the job, and srun then refuses to start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.10%. Comparing base (5c25d22) to head (b64c21e).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #222   +/-   ##
=======================================
  Coverage   72.10%   72.10%           
=======================================
  Files          22       22           
  Lines        2370     2370           
=======================================
  Hits         1709     1709           
  Misses        661      661           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants