promptinject : SKIP

[bouronthierry]

I'm using Garak with local llama.cpp model ( envs_python % python -m garak --model_type ggml --model_name "/Users/xx/Models_llm/Llama-3.2-3B-Instruct-Q4_0.gguf" --probes promptinject . The test is running 256 iterations at the end it display promptinject.AttackRogueString: SKIP. Do you know why? how can I solve it?

[Advait251206]

promptinject.AttackRogueString: SKIP in Garak usually doesn’t indicate a crash—it means the probe could not find a valid attack path for that model/backend combination in the current configuration, so it is being intentionally skipped.

In your specific setup (local llama.cpp / GGUF model), there are a few common reasons this happens:

---

1. The probe requires capabilities your backend doesn't expose

promptinject probes often assume:

• system / developer / user role separation
• structured chat templates (like OpenAI / ChatML style)
• stable instruction hierarchy

But ggml/llama.cpp models:

• often flatten prompts into a single string
• may not preserve role boundaries
• may not expose controllable system prompt injection surfaces

👉 Result: Garak cannot reliably construct the “rogue string attack” scenario → SKIP

---

2. Missing or incompatible chat template

If your model is a Llama 3.x instruct GGUF, but Garak is not applying the correct chat template, then:

• prompts are malformed
• injection structure cannot be inserted correctly
• probe logic aborts

This is especially common with:

--model_type ggml

where template handling is minimal or inferred.

---

3. The probe is intentionally gated

Some promptinject sub-probes only run if:

• tool calling is supported
• multi-turn context injection is possible
• or model supports specific formatting assumptions

If those checks fail → SKIP (not FAIL)

---

4. Iterations finishing doesn’t mean execution succeeded

The “256 iterations” you see means:

• Garak still ran the harness loop
• but individual probe targets were skipped internally

So you’re seeing:

harness executed ✔
probe executed ❌ (skipped internally)

---

How to fix / improve this

Option 1: Use a proper chat-format backend

Instead of raw ggml, try:

• --model_type llama.cpp
• or a server-based backend (llama.cpp server / vLLM / OpenAI-compatible endpoint)

This allows proper role formatting.

---

Option 2: Force correct chat template

If using llama.cpp:

• ensure --chat_template matches Llama 3 Instruct format
• or use a Garak-supported template override

---

Option 3: Switch model interface

Best compatibility for promptinject:

• OpenAI API-compatible endpoint
• vLLM OpenAI server mode
• Ollama (sometimes better template handling than raw ggml)

---

Option 4: Check probe applicability

Run:

python -m garak --list_probes

and check if promptinject.AttackRogueString has backend requirements.

---

Key takeaway

This is not a failure of your model—it’s a compatibility gap between:

• Garak’s prompt-injection probe assumptions
  vs
• llama.cpp GGUF runtime constraints

So SKIP here is expected behavior in many GGUF setups.

---

If you want, I can suggest a minimal configuration that makes promptinject fully run on local Llama 3 GGUF without skips.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206