feat: xAI (Grok) voice provider suite — STT, TTS, Realtime in both SDKs

[nicolotognoni]

Summary

• Adds the full xAI (Grok) voice provider suite to both SDKs: Realtime (Grok Voice Agent API), streaming + batch STT, and TTS with custom-voice cloning — every voice API xAI currently ships.
• Wires official xAI pricing into cost metrics: Realtime $0.05/min ($3.00/hr), TTS $15.00/1M chars, STT $0.20/hr streaming / $0.10/hr batch.
• Adds Mintlify docs: 6 new provider pages (both SDK trees), nav registration, metrics pricing tables, and STT/TTS/Engines landing-page updates.

Implementation

• Realtime — XaiRealtime engine + XaiRealtimeAdapter: xAI's Voice Agent API (wss://api.x.ai/v1/realtime, default model grok-voice-latest, voice eve) is OpenAI-Realtime-GA protocol-compatible, so the adapter subclasses the GA adapter and inherits audio streaming, barge-in, session handshake, and the entire tool-calling bridge (including built-in transfer_call/end_call). The TypeScript GA adapter gained a small realtimeBaseUrl() hook (pure refactor, behavior byte-identical) so protocol-compatible providers can override the endpoint; Python overrides the URL class attribute. xAI-specific session knobs are exposed opt-in: reasoning_effort ("high"/"none"), VAD threshold/prefix_padding_ms/idle_timeout_ms, ASR language_hint/keyterms, output speed, pronunciation replace map, session resumption, and raw server_tools passthrough (xAI server-side web_search, x_search, mcp, file_search). Input transcription model defaults to grok-transcribe. v1 inherits the proven GA PCM24 transcode path; native G.711 passthrough (xAI supports audio/pcmu @ 8 kHz) is flagged as a follow-up latency optimization.
• STT — XaiSTT (provider_key "xai"): streaming WebSocket adapter (wss://api.x.ai/v1/stt, config via URL query params, raw binary frames, transcript.created handshake gating, chunk-final/utterance-final mapping, finalize support, Smart Turn end-of-turn detection, repeatable keyterm biasing, diarization, filler words). Defaults pcm@16000 to match the pipeline's decoded feed (consistent with Deepgram/Soniox). Plus a batch transcribe() / xaiTranscribe() REST helper (multipart with file-last ordering, word-level timestamps, ITN formatting, URL-or-bytes input).
• TTS — XaiTTS (provider_key "xai_tts"): REST one-shot streaming synthesis (POST /v1/tts), 26-voice roster (eve default), language="auto" default, for_twilio() (G.711 µ-law 8 kHz) / for_telnyx() (PCM16 16 kHz) carrier presets, speed, optimize_streaming_latency, text_normalization, MP3 bit_rate; plus create_custom_voice() / xaiCreateCustomVoice() for POST /v1/custom-voices voice cloning (≤120 s reference clip).
• Pricing/metrics: three new DEFAULT_PRICING entries; calculate_realtime_cost / calculateRealtimeCost gain optional provider and duration arguments to support per-minute realtime billing (token path unchanged — regression-tested); metrics accumulator meters xai_realtime by session duration. The xAI $0.004/text-input-message charge is documented as a pricing constant but not metered in v1.
• Wiring: engine marker → client engine unpacking → stream-handler adapter selection → telephony key plumbing (xai_key), telemetry vendor allowlist + aliases, root exports, pyproject.toml [xai] extra (aiohttp).
• Files: libraries/python/getpatter/providers/xai_{stt,tts,realtime}.py, libraries/typescript/src/providers/xai-{stt,tts,realtime}.ts, engines/xai.*, stt/xai.*, tts/xai.*, pricing/metrics/telemetry/client/server edits, 6 docs pages + docs.json + metrics/landing pages, CHANGELOG.
• Deps: none added to TS; Python gets an optional [xai] extra reusing aiohttp (same pin as the soniox extra).

Breaking change?

No. All new surface is opt-in with safe defaults; existing providers, pricing entries, and the token-based realtime cost path are unchanged (existing OpenAI Realtime tests pass untouched).

Beta notes / known limitations

• Marked Beta: validated against the xAI API spec and mocked protocol tests; not yet exercised on a live phone call.
• Caller-side transcript display does not populate in v1 (xAI emits input_audio_transcription.updated rather than .completed); audio, responses, barge-in, and tool calling are unaffected. Documented in the provider page.
• xAI WS streaming TTS variant deferred (pipeline consumes one-shot synthesis; precedent: elevenlabs_ws exists as a separate provider — a future xai_ws_tts could mirror it).
• Pre-existing repo-wide gap (not introduced here): TS package.json exports only maps ".", so namespaced subpath imports shown in provider docs resolve for types but not at runtime; docs lead with the flat aliases.

Test plan

• Python: pytest tests/ — 3124 passed, 8 skipped, 2 xfailed (34+ new xAI tests: WS handshake ordering, is_final/speech_final mapping, finalize, multipart ordering, session-config grafting, pricing math incl. per-minute realtime + unchanged token path)
• TypeScript: npm test — 2542 passed (45+ new xAI tests incl. GA-adapter refactor regression) + npm run lint (tsc clean) + npm run build (success)
• /parity-check clean — dedicated parity audit: public surface, defaults, pricing values, session payloads, wire protocol, telemetry all aligned (3 gaps found and fixed during the audit, with regression tests)
• E2E smoke — N/A in CI (requires live xAI account + phone call; providers are Beta until a live call validates them)

Docs updates

• docs/{python,typescript}-sdk/providers/xai-stt.mdx, xai-tts.mdx, xai-realtime.mdx (new, both trees)
• docs/docs.json (6 nav insertions), docs/{python,typescript}-sdk/metrics.mdx (pricing tables), docs/{python,typescript}-sdk/{stt,tts,engines}.mdx, docs/concepts.mdx

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
patter-06b046ce	🟢 Ready	View Preview	Jul 13, 2026, 11:11 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.