AI chat assistant: site-aware conversational guide

[silverbackdan]

Goal

Build an AI-powered chat assistant embedded in the docs/marketing site that can discuss CWA in a relatable, conversational way — similar to how NotebookLM works after ingesting a knowledge base.

What it should do

• Answer questions about whether CWA is right for the user's project
• Discuss how the module would work for their specific use case
• Draw on both the advertising/marketing pages (about, built-for-business, etc.) and the full technical documentation
• Help non-technical stakeholders understand the value proposition
• Help developers understand how to approach their implementation

Knowledge sources

• /llms-full.txt — full docs content (auto-generated, always up to date)
• /llms.txt — structured index
• Marketing/about page YAML content (content/*.yml)
• Potentially the GitHub READMEs / changelogs of the API bundle and Nuxt module

Approach (from earlier design discussion)

The owner has additional context from a prior Claude session on the planned implementation approach. Key notes:

• Embed the chat UI on the docs/marketing site (not a separate tool)
• Ingest the full site content so the AI has CWA-specific knowledge, not just generic answers
• The tone should be relatable and advisory, not a dry FAQ bot
• Should handle both "is this right for me?" pre-sales questions and "how do I do X?" developer questions

Foundation already in place

• GET /llms.txt — structured page index, auto-updated on deploy
• GET /llms-full.txt — full concatenated docs text, auto-updated on deploy

These can be fed directly to an AI system at startup or used as retrieval context.

Approach (decided)

Retrieval + Claude Haiku 4.5, behind an /api/chat edge function. Full pricing comparison and prompt design in the comments.

• Provider: Claude Haiku 4.5 ($0.012/turn → **$6/month** at 500 turns). Gemini 3.1 Flash-Lite is 4× cheaper ($1.55/mo) but at our traffic that gap is ~$4.50/month, and Google's model churn is faster (2.5 Flash-Lite retires Oct 2026; 3.1/3.5/3.6 already shipped) for a widget nobody is assigned to maintain. Note Haiku undercuts Gemini's Flash tiers — "Gemini is cheaper" only holds at Flash-Lite. Both vendors' paid tiers exclude training on inputs, so there is no privacy trade either way.
• Retrieval, not full-context. llms-full.txt is ~165–190k tokens. Stuffing it costs ~$0.45 per cold prompt-cache write on a 5-min TTL, so low traffic makes it worse, not cheaper — and on Gemini it's far worse still, as context caching there bills $1.00/hour in storage ($720/month) on top. Retrieving ~8k tokens avoids the problem entirely; lexical/BM25 over our own docs needs no embeddings service.
• NotebookLM cannot be the backend. No consumer API; the Gemini Notebook Enterprise API (v1alpha) is notebook/source management only — no query or chat endpoint. The only free NotebookLM path is publishing a public shared notebook and linking out.
• Free tiers rejected for a public endpoint. Volume is fine at our traffic, but the Gemini free tier trains on submitted content and human reviewers may read inputs/outputs — the exposure is visitor questions, not our public docs. Groq (no retention by default, ZDR on all tiers) was viable and set aside. Being FOSS doesn't disqualify us from any free tier; commerciality was never the constraint.
• Self-hosting costs more, not less — ~$100–300/month for a GPU VPS vs ~$6/month here, and the docs site prerenders to Vercel with nowhere to run inference. Retrieval is the only piece worth self-hosting.
• Persona: conversational, metaphor-led for non-technical visitors, with the grounding rule split explicitly — metaphors are the model's to invent, facts are not — to stop it improvising product behaviour from other CMSs. Starter system prompt and seed metaphors in the comments.

Cost guards required regardless: cap turns and message length, rate-limit per IP, degrade to plain search when limits trip.

Next steps

1. Decide on the AI provider / embedding approach — done, see above and the findings comment
2. Design the chat UI component
3. Decide where it lives — floating button on all pages, dedicated /ai page, or both
4. Handle the marketing pages content (currently YAML, needs to be included in the knowledge base)

[silverbackdan]

Findings: NotebookLM can't be the backend, and the free tiers are viable

Researched the two open questions from Next steps #1 (AI provider / RAG vs full-context). Summary: NotebookLM is a dead end as an integration, but the free tiers are a genuine fit given our traffic.

NotebookLM — no usable integration

• No consumer/public API. Google has acknowledged demand and says one is planned; nothing self-serve exists today.
• Gemini Notebook Enterprise has an API, but it cannot answer questions. It's v1alpha (preview) and covers notebooks.create / .get / .listRecentlyViewed / .batchDelete / .share, plus podcast generation. There is no query or chat endpoint — you can create a notebook and load sources programmatically, then have no supported way to ask it anything. It also needs Gemini Enterprise licences.
• An unofficial reverse-engineered client exists (notebooklm-py) driving the private endpoints. Fine for scripting; not appropriate behind a public page — ToS exposure, needs a logged-in Google session, breaks whenever Google changes internals.
• The one free NotebookLM path: publish a public shared notebook and link to it. Zero build, zero cost, and it is literally the thing that impressed us. Trade-off: off-site, Google-branded, unstyleable, outside our analytics. Reasonable stopgap, not a product feature.

Free tiers — volume is a non-issue; data policy is the real differentiator

Our traffic is low, so every free tier below has ample headroom. What actually separates them is what happens to visitor questions.

Provider	Free allowance	Data posture
Groq	~30 RPM / ~14,400 req/day (org-level)	No retention by default; Zero Data Retention available to all customers; no free/paid split
Cloudflare Workers AI	10,000 Neurons/day	Training policy not stated on the pricing page
Google Gemini	~250 req/day (2.5 Flash), ~1,000/day (Flash-Lite)	Trains on submitted content; human reviewers may read inputs and outputs

The Gemini free-tier clause is the important finding, straight from the official terms:

"Google uses the content you submit to the Services and any generated responses to provide, improve, and develop Google products and services"

and

"human reviewers may read, annotate, and process your API input and output."

The paid tier explicitly does not do this. This matters more than our licence does: the data at risk is our visitors' questions — a prospective user describing their project, their stack, sometimes their client — not our docs, which are public anyway. Using the Gemini free tier for a public chat endpoint would need a visible privacy notice on the widget. That's a UX and disclosure cost, not a legal blocker.

Worth noting on the commercial question: Google's terms frame AI Studio / the Gemini API as "for developers building with Google AI models for professional or business purposes, not for consumer use" — they don't prohibit production deployment, and being FOSS doesn't disqualify us. The free-tier constraint that actually bites simply isn't commerciality.

Also checked and rejected: self-hosting is more expensive than the API, not less. A GPU VPS is roughly $100–300/month against ~$24/month for a hosted small model; CPU-only inference on a 7–8B model is seconds-per-token, which is a poor first impression for a pre-sales visitor. Our existing Docker stack doesn't help — the docs site is prerendered to Vercel, with nowhere to put an inference container. The one self-hosted piece worth building is retrieval, which needs no GPU and no API.

Recommendation

Groq free tier + lexical retrieval, behind an /api/chat edge function:

1. Retrieval, not full-context. llms-full.txt is ~165–190k tokens. Stuffing it costs ~$0.45 per cold prompt-cache write, and on a low-traffic site the cache (5-min TTL) is cold nearly every time — low traffic makes full-context worse, not cheaper. Retrieving ~8k relevant tokens sidesteps this, and lexical/BM25 over our own docs needs no embeddings service. We already generate a search index.
2. Groq for inference — most generous free limits, best data posture of the three, fast by design, and its OpenAI-compatible API keeps the swap cost low.
3. Keep the guards regardless of price. A public unauthenticated endpoint is scriptable: cap turns per conversation and characters per message, rate-limit per IP, and degrade to plain search when limits trip. Free tiers fail closed with 429s rather than a surprise bill, which is a point in their favour.
4. Upgrade path if quality disappoints on the "is CWA right for my project?" questions — those are the ones needing real judgement. Claude Haiku 4.5 with the same retrieval design is $0.012/turn ($24/month at 2,000 turns).

Rate-limit figures are indicative: Google's official rate-limit page no longer publishes per-model free-tier numbers (it defers to the AI Studio dashboard), and the free quotas were cut in Dec 2025 — so treat headroom as generous but not contractual.

Still open

• Chat UI design and placement (floating button vs /ai page vs both) — Next steps Docs: add image placeholder callouts throughout guides #2/Docs: add video placeholder callouts in step-by-step guide sections #3, unaffected by the above.
• Marketing YAML (content/*.yml) is not in llms-full.txt; the retrieval corpus needs to include it — Next steps Docs: improve file structure summary visuals #4.

[silverbackdan]

Decision: Claude Haiku 4.5 — and full provider pricing

Superseding the provider recommendation in my previous comment. Groq remains technically viable (best free limits, no-retention default) but we're going with Claude Haiku 4.5 on the paid tier. Reasoning below, with the full pricing comparison that informed it.

Pricing, costed for our actual design

Per turn assumes ~10k retrieved context + ~400 output tokens. Gemini figures are from Google's official pricing page.

Model	In $/M	Out $/M	Per turn	~500 turns/mo	~2,000 turns/mo
Gemini 3.1 Flash-Lite	0.25	1.50	$0.0031	$1.55	$6.20
Gemini 3.5 Flash-Lite	0.30	2.50	$0.0040	$2.00	$8.00
Claude Haiku 4.5	1.00	5.00	$0.0120	$6.00	$24.00
Gemini 3.6 Flash	1.50	7.50	$0.0180	$9.00	$36.00
Gemini 3.5 Flash	1.50	9.00	$0.0186	$9.30	$37.20
Claude Sonnet 5 (intro)	2.00	10.00	$0.0240	$12.00	$48.00
Gemini 3.1 Pro Preview	2.00	12.00	$0.0248	$12.40	$49.60

Four things worth recording:

• Haiku sits between Flash-Lite and Flash, not above Flash. Gemini's Flash tiers (3.5, 3.6) cost ~50% more per turn than Haiku. "Gemini is cheaper" only holds at the Flash-Lite tier; one step up and it inverts. So our upgrade path from a cheap model is Haiku at $6/mo, not Gemini Flash at $9/mo.
• Gemini context caching carries an hourly storage fee — $1.00/hr on Flash tiers, $0.50–$1.80/hr on 3.1 Flash-Lite. Held continuously that's ~$720/month to store a cache, dwarfing inference. This kills any "cache llms-full.txt" approach on Gemini outright. Anthropic caching has no storage fee (just a 1.25×/2× write premium), so the same idea merely fails on the 5-min TTL. Either way: retrieval, not caching — but the trap is far more expensive on Google's side.
• Gemini 3.1 Pro Preview has no free tier; free tier covers Flash and Flash-Lite only.
• Batch and "flex" tiers are 50% off across all Gemini models. Flex is a lower-priority synchronous tier — genuinely interesting for a docs widget where a second of extra latency is invisible.

Sonnet 5's $2/$10 is introductory pricing ending 31 Aug 2026 (then $3/$15 → ~$0.034/turn) — don't budget against it.

Why Haiku over Gemini 3.1 Flash-Lite despite the ~4× price gap: at our traffic the absolute difference is ~$4.50/month, which doesn't justify optimising. Against that, Google's model churn is faster (2.5 Flash-Lite retires Oct 2026; 3.1, 3.5 and 3.6 lines already shipped) and this is a side-project widget nobody is assigned to maintain — a provider that deprecates more slowly is worth a few pounds a month. Anthropic's paid tier also doesn't train on inputs, same as Gemini's paid tier, so there is no privacy trade either way.

---

Prompt design: conversational, with metaphors

The goal is a relatable advisor, not an FAQ bot — so the persona work matters as much as the provider.

The tension to design around

The RAG prompt says "only use the retrieved docs." The persona prompt says "explain with metaphors." A metaphor is by definition not in the docs, so this produces one of two failure modes:

1. The model refuses to be creative because nothing in context licenses it → dry FAQ bot; or
2. It reads the metaphor licence as general permission to improvise, and smuggles in facts half-remembered from WordPress or Strapi.

The second is the dangerous one — a confident, plausible, wrong answer about our own product. Fix: split the permission explicitly. Metaphors are the model's to invent; facts are not.

Starter system prompt

You are the CWA assistant, embedded on the Component Web App docs and marketing
site. CWA is free and open source.

## Who you're talking to
Two kinds of visitor — work out which within a message or two:
- Evaluating: "is this right for my project?", "what is this?". Often non-technical.
  They need the shape of the idea, not the API.
- Building: "how do I do X?", "why isn't my component rendering?". They want
  specifics, code, and a link to the exact page.
If you can't tell, assume evaluating and offer to go deeper.

## How to explain things
Lead with a concrete everyday comparison, then the real term, then link the docs.
Metaphors are for structure and relationships — never for facts, numbers,
versions, or API names.
One metaphor per answer. Don't stack them or stretch one past its usefulness.
If a metaphor would mislead about something that matters, drop it and say the
plain thing.

## Grounding
Everything factual comes from the documentation excerpts provided. If they don't
cover it, say so and point at the nearest relevant page. Do not fill gaps from
general knowledge about other CMSs or frameworks.
The metaphors are yours to invent. The facts are not.

## Tone
Warm, direct, plain. Short paragraphs. No bullet walls for a simple question.
Don't oversell — if CWA is a poor fit for what they describe, say so. That earns
more trust than a pitch. You're helping them decide, not closing them.

The "say so if it's a poor fit" line is load-bearing: an assistant that never says "probably not for you" gets ignored; one that occasionally does gets believed.

Seed metaphors

Rather than leaving the model to invent from scratch, seed these — they're accurate to the actual architecture:

Concept	Comparison
Layout with component groups	A picture frame with labelled slots — editors swap the pictures without touching the frame
Layout → page → group → component	Nested boxes; each level only knows about the one inside it
Dynamic pages (PageData)	One blueprint, many houses — a single blog template serves every article
Draft vs published	A shop window versus the stockroom; rearrange out of sight, then put it on display
Inline editing	Editing the live page directly, rather than filing a change request against it
API-first split	The API is the kitchen, the Nuxt app is the dining room — same meal, different concerns

Explicitly forbid "it's like WordPress." It's the comparison every visitor reaches for and it sets exactly the wrong expectations about the API-first split and the component model.

Gotchas

• Prompt caching won't fire on Haiku 4.5 — its minimum cacheable prefix is 4,096 tokens, and a persona prompt won't reach that, so cache_control silently no-ops. Irrelevant at ~$6/month, but don't wire it up expecting savings.
• Tone is what actually needs evaluating. Cost and correctness can be reasoned about on paper; whether it reads like a helpful colleague can only be judged from transcripts. Run ten real questions — five "is this right for me?", five "how do I do X?" — and read the answers before shipping.