Skip to content

test(qdc): run llama_cpp LLM + VLM QDC matrix on cpu/gpu/npu #1308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Mengsheng Wu (mengshengwu) wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

test(qdc): run llama_cpp LLM + VLM QDC matrix on cpu/gpu/npu #1308

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
db50c02
test(qdc): run llama_cpp LLM + VLM QDC matrix on cpu/gpu/npu
mengshengwu Aug 7, 2026
7971c96
ci(qdc): widen path filter from tests/qdc/ to tests/
mengshengwu Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/_qdc-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,9 @@ jobs:
if: inputs.gate_on_changed_files
uses: ./.github/actions/changed-files
with:
include: '^(sdk|third-party|tests/qdc)/|^\.github/workflows/_qdc-test\.yml$'
# QDC harness bundles the full tests/ tree onto the device, so any
# tests/** change (not just tests/qdc/**) affects the on-device run.
include: '^(sdk|third-party|tests)/|^\.github/workflows/_qdc-test\.yml$'

- name: Decide skip
id: skip
Expand Down
5 changes: 3 additions & 2 deletions tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ model-manager-pull check that runs first:
| `test_vlm_quality_keywords` | Golden-retriever caption must match one of the canonical keywords. |
| `test_mtp_multi_turn` | (llama_cpp only) same "Alice" convo with `spec_type='draft-mtp'`. |

Backends per plugin: `llama_cpp` on `cpu` + `npu`; `qairt` on `npu` only.
Backends per plugin: `llama_cpp` on `cpu` + `gpu` + `npu`; `qairt` on `npu` only.
Model-manager pull failures are treated as FAIL, not SKIP, so a broken
download surfaces as a red CI leg instead of a silent green skip.

Expand All @@ -44,8 +44,9 @@ The conftest auto-tags items by location and `device_map` value:
| `llama_cpp` | items in `tests/test_llama_cpp.py` |
| `qairt` | items in `tests/test_qairt.py` |
| `device_cpu` | parametrised with `device_map='cpu'` |
| `device_gpu` | parametrised with `device_map='gpu'` |
| `device_npu` | parametrised with `device_map='npu'` |
| `snapdragon` | any `device_map='npu'` cell (auto-applied) |
| `snapdragon` | any `device_map` in {`gpu`, `npu`} cell (auto-applied) |
| `llm` / `vlm` | applied per-test via `@pytest.mark.llm` / `.vlm` |

`snapdragon`-marked and `qairt` items skip automatically unless
Expand Down
4 changes: 3 additions & 1 deletion tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,11 @@

_DEVICE_MARKER = {
'cpu': 'device_cpu',
'gpu': 'device_gpu',
'npu': 'device_npu',
}
_SNAPDRAGON_DEVICES = {'npu'}
# GPU uses the Snapdragon OpenCL backend, so it needs real hardware just like NPU.
_SNAPDRAGON_DEVICES = {'gpu', 'npu'}


def _is_snapdragon_host() -> bool:
Expand Down
1 change: 1 addition & 0 deletions tests/pytest.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ markers =
llm: text-only generation
vlm: vision-language generation
device_cpu: cpu backend (works anywhere)
device_gpu: OpenCL GPU backend (Snapdragon only)
device_npu: HTP NPU backend (Snapdragon only)
snapdragon: requires real Snapdragon host (gated by GENIEX_DEVICE_TEST=1)
norecursedirs = qdc
9 changes: 5 additions & 4 deletions tests/test_llama_cpp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@
VLM_QUALITY_TEMPERATURE,
)

_LLM_BACKENDS = ['cpu', 'npu']
_VLM_BACKENDS = ['cpu', 'npu']
_LLM_BACKENDS = ['cpu', 'gpu', 'npu']
_VLM_BACKENDS = ['cpu', 'gpu', 'npu']


def test_model_manager_pull(llama_cpp_llm_paths, llama_cpp_vlm_paths, llama_cpp_mtp_paths):
Expand Down Expand Up @@ -80,11 +80,12 @@ def test_llm_multi_turn(llama_cpp_llm_paths, device_map):


@pytest.mark.vlm
def test_vlm_multi_turn(llama_cpp_vlm_paths, test_image):
@pytest.mark.parametrize('device_map', _VLM_BACKENDS)
def test_vlm_multi_turn(llama_cpp_vlm_paths, test_image, device_map):
with geniex.AutoModelForVision2Seq.from_pretrained(
LLAMA_CPP_VLM_MODEL,
precision=LLAMA_CPP_VLM_PRECISION,
device_map='cpu',
device_map=device_map,
) as vlm:
history = [
{
Expand Down
Loading