Documentation and deployment repository for running LLMs locally on CPU hardware.
- Concept docs:
README.md,prefill.md,software-stack.md - Deployment:
deployments/— Ollama + oh-my-pi via Docker Compose (CPU-only, Gemma 4 26B Q4) - Import:
import/— external documents for incoorporation into the project
Runs Gemma 4 26B MoE (Q4_K_M, ~18 GB RAM) via Ollama on a CPU-only machine.
The agentic harness is oh-my-pi (omp), running in a separate Bun-based container.
| Task | Command |
|---|---|
| One-time setup | bash scripts/setup.sh |
| Start Ollama | docker compose up -d ollama |
| Pull / re-register model | bash ollama/scripts/pull-model.sh |
| Start omp (interactive) | bash omp/scripts/start-omp.sh |
ollama/modelfiles/gemma4.Modelfile— sampling params,num_ctx, stop sequencesomp/config/models.yml— Ollama provider + model definition for ompomp/config/config.yml— omp agent roles, compaction, retry settings
num_ctx: 65536is the calibrated sweet spot. KV cache: 32K ≈ 3–4 GB, 65K ≈ 6–7 GB, 128K ≈ 12–16 GB. Host has ~10–14 GB free after weights load.- Never add GPU device config to
compose.yaml— CPU-only mode; adding GPU resources breaks the setup. - Q8 variant OOMs on 32 GB: Q8 weights alone require ~32 GB. Only viable on ≥48 GB hosts.
auth: noneinmodels.ymlis required — an emptyOLLAMA_API_KEYenv var is not equivalent; omp will silently filter out the provider without it.reasoning: falseinmodels.ymlis intentional — Gemma 4 thinking is prompt-controlled (prepend<|think|>to system prompt), not via API extension flags that Ollama doesn't support.- CPU prefill latency: Expect 2–6 minutes before the first output token on large inputs. See
prefill.md.
| Topic | File |
|---|---|
| Stack layers, inference engine comparison | software-stack.md |
| Prefill vs decode, KV cache sizing | prefill.md |
| Prompt anatomy, agentic harness concepts | README.md |
| Deployment overview | deployments/README.md |