How to use a Remote Embedder that is through Llamacpp API with Haystack

[IsmailCSV]

Hello, I had previously created a pipeline using a local embedder that I just downloaded through Huggingface now I am trying to use an embedder that is hosted on a server through Llamacpp. I've done tons of research and have tried to utilize ChatGPT, but I cannot find an answer. I am hoping there is anyone here that could help or guide me in the right direction on how to achieve this.

[sjrl]

Hey @IsmailCSV it looks like the Llama.CPP integration with Haystack only supports the ChatGenerator currently.

I'd recommend opening an issue in this repo https://github.com/deepset-ai/haystack-core-integrations requesting support for embedding models hosted through Llama.cpp!

[gyanu2507]

llama.cpp's server speaks OpenAI embeddings. Point Haystack at that, not at a local Hugging Face embedder class.

from haystack.components.embedders import OpenAITextEmbedder, OpenAIDocumentEmbedder

embedder = OpenAITextEmbedder(
    api_base_url="http://HOST:8080/v1",
    api_key="sk-not-used",
    model="your-embed-model-name",
)

Same idea for OpenAIDocumentEmbedder on the indexing pipeline. The model string has to match whatever llama-server registered (often the filename).

HuggingFaceAPITextEmbedder is for HF Inference. SentenceTransformersTextEmbedder loads weights in-process. Neither talks to llama.cpp.

Check with curl http://HOST:8080/v1/embeddings first. If that 404s, the server was started without embeddings / with a generative GGUF only.

[kasimlohar]

llama.cpp's server speaks the OpenAI embeddings API, so point Haystack's OpenAI embedders at it — verified against haystack/components/embedders/openai_text_embedder.py, which accepts api_base_url and forwards it as the OpenAI client's base_url.

One correction to the snippet above: api_key must be a Secret, not a plain string (a plain "sk-not-used" fails at resolve_value()). Use:

from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
from haystack.utils import Secret

# Query side
query_embedder = OpenAITextEmbedder(
    api_base_url="http://HOST:8080/v1",
    api_key=Secret.from_token("sk-no-key-needed"),
    model="your-embed-model-name",  # must match the model id llama-server registered
)

# Indexing side (same server, document variant)
doc_embedder = OpenAIDocumentEmbedder(
    api_base_url="http://HOST:8080/v1",
    api_key=Secret.from_token("sk-no-key-needed"),
    model="your-embed-model-name",
)

Pre-checks before running:

• curl http://HOST:8080/v1/models — confirm the model id and that the server was started with embeddings enabled (a generative-only GGUF / no --embedding flag gives 404 on /v1/embeddings).
• Query and index embedders must point at the same model — mismatched dimensions silently break retrieval.
• SentenceTransformers* loads weights in-process and HuggingFaceAPI* talks to HF Inference — neither talks to llama.cpp, so don't use those here.

Longer term, per @sjrl above, native embedding support doesn't exist in the llama_cpp integration (chat only) — worth opening a feature issue on haystack-core-integrations if you want first-class support instead of the OpenAI-compat workaround.