coach: on-device AI Coach — bundled local LLM via llama.cpp (iOS)#553
Open
hkuehl wants to merge 1 commit into
Open
coach: on-device AI Coach — bundled local LLM via llama.cpp (iOS)#553hkuehl wants to merge 1 commit into
hkuehl wants to merge 1 commit into
Conversation
- New .onDevice provider backed by in-process llama.cpp (pinned b9947 xcframework via the LlamaCpp SPM package), first-run download of Llama-3.2-3B-Instruct Q4_K_M, streaming token generation. iOS default provider; hidden on macOS. - ModelDownloadManager (streaming SHA-256 verify), ModelCatalog, OnDeviceClient, LlamaEngine actor, CoachView setup card, increased-memory entitlement. - Fix: clear the KV cache before each prefill so reused-context turns start at position 0 (cumulative positions were overrunning n_ctx and killing the coach after ~3 turns; also fixes attending to duplicated context on earlier turns). - Fix: the Coach toolbar gear now opens a settings sheet (provider / model / key) instead of a "Disconnect" action that was a silent no-op for the on-device provider — so settings stay reachable once the model is downloaded. - Performance pass: coalesce streaming UI updates (~16 Hz flush, not a full MarkdownUI re-parse per token — was O(n^2) over the reply); pin n_ubatch, enable Flash Attention, and set n_threads on the llama context; size the chat-template buffer exactly (no silent prompt truncation); resumable model download (continues a dropped transfer instead of restarting ~2 GB). - License compliance: NOTICE entries for llama.cpp (MIT) and the downloaded Meta Llama 3.2 model (Community License, not redistributed); "Built with Llama" attribution + license/AUP links in the coach setup UI (palette-tinted); model download URL pinned to an immutable Hugging Face commit revision (sha256 still fail-closed). Bump MARKETING_VERSION to 9.1.0 (new feature; SemVer MINOR). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a fifth AI Coach provider,
.onDevice, that runs a small LLM entirely on the iPhone via in-process llama.cpp (Metal) — no API key, and after a one-time model download, no network. It becomes the default, front-and-centre coach on iOS; the existing cloud BYOK providers (OpenAI / Anthropic / Gemini / Custom) are unchanged.ModelCatalog.swift).ggml-orgrelease b9947) as an SPM binary target with URL + checksum pinned (Packages/LlamaCpp).AIProviderClientprotocol, so the whole coach pipeline — context building, consent gating, chat history,CoachView— is reused. iOS-only (LlamaEngine/OnDeviceClientexcluded from the macOS target; provider hidden on macOS).Offline-by-design note
Coach inference is 100% on-device. The only network activity is the one-time, user-initiated download of the public model weights (opposite direction — no user data leaves the device). This is an extension of the already opt-in AI Coach, the project's existing single networked feature; the offline-first posture is preserved.
Memory & stability (iOS)
com.apple.developer.kernel.increased-memory-limitentitlement; ~6 GB-RAM device gate (ModelCatalog.deviceMeetsRequirements).n_ctx). Flash Attention enabled,n_ubatch/n_threadsset, and streaming UI updates coalesced to ~16 Hz (was an O(n²) MarkdownUI re-parse per token).Licensing / attribution
NOTICEupdated: llama.cpp / ggml (MIT) and the downloaded Llama 3.2 model (Meta Llama 3.2 Community License — not redistributed).Verification
NOOPiOSlocally against the real b9947 headers (theapp-buildworkflow is disabled by default, so this isn't covered by CI). The macOSStrandscheme compiles too (on-device path excluded there).StrandTests: provider gating, model catalog, download manager, streaming adapters).StrandDesigntokens. No BLE / sync-path changes. iOS-only — no Android code, so the cross-platform parity contract doesn't apply here.MARKETING_VERSIONbumped to9.1.0(new feature; SemVer MINOR).🤖 Generated with Claude Code