Skip to content

[QDP] Unify GPU-availability detection in tests; stub CUDA path should raise, not abort #1414

Description

@ryankert01

Context

#1321 lets qdp-core / _qdp build and import without the CUDA toolkit (it links stub CUDA Runtime symbols when nvcc is absent). This is the right capability, but it breaks a long-standing assumption baked into the test suite: "_qdp is importable" ⟺ "a GPU is usable." That assumption was true before (no toolkit → link failure → _qdp unimportable), so a single import check was enough to gate GPU tests.

After #1321, on a GPU-less runner _qdp imports fine (stubs), so the @pytest.mark.gpu tests ran against the stub engine and aborted the pytest workers (Fatal Python error: Aborted). #1321 patched the immediate breakage in testing/conftest.py (skip @pytest.mark.gpu when torch.cuda.is_available() is False) and marked one loader test, which is enough to make CI green — but the underlying detection strategy is fragmented and has a real gap. This issue tracks doing it properly.

Problems

  1. Fragmented "needs GPU/QDP" detection — no single source of truth. The suite currently expresses this three different ways:

    • @pytest.mark.gpu (~95 sites), now auto-skipped by conftest
    • inline if not torch.cuda.is_available(): pytest.skip(...) (~65 sites, e.g. testing/qdp/test_bindings.py)
    • import-only guards requires_qdp / _qdp_available() / _loader_available() (~110 sites) that still conflate "importable" with "GPU present"
  2. torch.cuda.is_available() is a proxy, not the truth. It reports torch's view of CUDA, not whether _qdp itself has a real runtime. Gap on fix(qdp-core): link cudart locally and stub FFI when toolkit is absent  #1321's headline scenario — a box with a GPU + PyTorch but no toolkit: _qdp is a stub build, yet torch.cuda.is_available() is True, so the GPU tests would run against the stub and abort again. _qdp exposes no "stub vs real CUDA" signal today.

  3. The stub path aborts instead of raising. fix(qdp-core): link cudart locally and stub FFI when toolkit is absent  #1321's description promises that calling a CUDA function on a no-toolkit build "surfaces a clean runtime error instead of failing at link time," but in practice the path SIGABRTs the process. test_synthetic_loader_batch_count even has a try/except RuntimeError that expects the clean behaviour. If the stub raised cleanly, the failures would be ordinary errors (no worker death) and far less test-side gating would be needed.

Proposed design

  • Expose _qdp's real-CUDA state to Python — e.g. a build-time flag / _qdp.cuda_available() distinguishing a stub build (and ideally probing for a device).
  • Add qumat_qdp.is_cuda_available() mirroring the existing is_triton_amd_available(), as the single source of truth.
  • Route the conftest auto-skip and the import-only guards through it; collapse the ~65 redundant inline torch.cuda.is_available() checks.
  • Make the stub CUDA path raise a clean RuntimeError rather than abort, fulfilling fix(qdp-core): link cudart locally and stub FFI when toolkit is absent  #1321's stated contract.

Not doing this in #1321 on purpose

#1321 is a contained cudart-linkage fix and is green. The above touches the Rust extension and 100+ test sites, so it belongs in its own PR.

Refs: #1321; conftest change in bcdd84264; the 10 tests that crashed were the @pytest.mark.gpu benchmark/bindings/high_fidelity tests plus test_synthetic_loader_batch_count.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions