Skip to content

Fix kernel-devel mismatch causing NVIDIA driver load failure - #32

Merged
skovili merged 2 commits into
rhos-vaf:mainfrom
skovili:fix/guest-kernel-devel-pin-mig
Aug 31, 2026
Merged

Fix kernel-devel mismatch causing NVIDIA driver load failure#32
skovili merged 2 commits into
rhos-vaf:mainfrom
skovili:fix/guest-kernel-devel-pin-mig

Conversation

@skovili

@skovili skovili commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Pin kernel-devel and kernel-headers to the running kernel version instead of installing the latest available from repos. This prevents DKMS from building the NVIDIA module for a different kernel than the one currently running.

Root cause: When the guest VM boots with kernel 5.14.0-570.12.1 but dnf installs kernel-devel-5.14.0-570.136.1 (latest from repos), DKMS builds the nvidia module for 570.136.1. When the guest tries to load the module, it fails because the running kernel is 570.12.1.

Impact: nvidia-smi returns rc=9 (NVML_ERROR_GPU_IS_LOST) because the driver can't load, breaking all GPU validation tests.

Fix: Detect the running kernel with uname -r and explicitly install kernel-devel-$(uname -r) to match.

Tested: Confirmed kernel mismatch on MIG CI job guest VMs at 192.168.122.222 (running 570.12.1, DKMS built for 570.136.1).

skovili and others added 2 commits August 30, 2026 22:24
Pin kernel-devel and kernel-headers to the running kernel version
instead of installing the latest available from repos. This prevents
DKMS from building the NVIDIA module for a different kernel than the
one currently running.

Root cause: When the guest VM boots with kernel 5.14.0-570.12.1 but
dnf installs kernel-devel-5.14.0-570.136.1 (latest from repos), DKMS
builds the nvidia module for 570.136.1. When the guest tries to load
the module, it fails because the running kernel is 570.12.1.

Impact: nvidia-smi returns rc=9 (NVML_ERROR_GPU_IS_LOST) because the
driver can't load, breaking all GPU validation tests.

Fix: Detect the running kernel with uname -r and explicitly install
kernel-devel-$(uname -r) to match.

Tested: Confirmed kernel mismatch on MIG CI job guest VMs at
192.168.122.222 (running 570.12.1, DKMS built for 570.136.1).

Related-Issue: MIG vGPU CI job failures since Aug 2026

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 31, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: d6f5618f-dd6a-4715-8d11-b571ed686d53

📥 Commits

Reviewing files that changed from the base of the PR and between 7080e19 and 53a3584.

📒 Files selected for processing (1)
  • gpu-validation/tasks/setup.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes
    • Improved GPU validation setup by installing kernel development and header packages that match the currently running kernel.
    • Helps ensure DKMS installation and validation work reliably across supported environments.

Walkthrough

The setup task captures the running kernel version and installs DKMS dependencies that match it.

Changes

RHEL MIG kernel dependencies

Layer / File(s) Summary
Match dependencies to the running kernel
gpu-validation/tasks/setup.yaml
The setup task records uname -r and installs version-matched kernel-devel and kernel-headers packages with DKMS.

Estimated code review effort: 2 (Simple) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 53a35

The change pins kernel development packages to the running kernel to prevent NVIDIA module mismatches; no actionable merge-blocking risk remains.

Suggested reviewers: miguelcarpio

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the kernel-devel mismatch and its impact on NVIDIA driver loading.
Description check ✅ Passed The description directly explains the kernel mismatch, root cause, fix, impact, and testing for the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@rlandy

rlandy commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

lgtm

@rlandy rlandy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@skovili
skovili merged commit 7efa368 into rhos-vaf:main Aug 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants