feat(sonality): token budgets, request caching, API auth, and Telegram isolation#84
Merged
Okm165 merged 141 commits intoJun 10, 2026
Conversation
…ings Add granular max_tokens configuration for different LLM task types: - ESS_MAX_TOKENS (512): ESS classification calls - STRUCTURED_JSON_MAX_TOKENS (256): routing, sufficiency, boundary checks - EXTRACTION_MAX_TOKENS (1024): knowledge/feature extraction - RERANK_MAX_TOKENS (512): episode reranking Add Neo4j connection pool settings: - NEO4J_MAX_POOL_SIZE (50) - NEO4J_CONNECTION_TIMEOUT (30.0s) Add LLM retry/backoff configuration: - LLM_MAX_RETRIES (3) - LLM_BACKOFF_BASE (2.0) Add additional settings: - EMBEDDING_CACHE_SIZE (10000): max cached query embeddings - LLM_CONCURRENCY (1): parallel LLM calls for multi-GPU - CHAT_INPUT_TOKEN_BUDGET (28000): input token limit for trimming - HTTP_API_KEY: optional API authentication - EPISODE_CONTENT_LIMIT, REFLECTION_USER_SLICE, REFLECTION_AGENT_SLICE - ESS_TIMEOUT, FORGETTING_CANDIDATE_LIMIT, RETRIEVAL_DECOMPOSE_MIN_WORDS Simplify .env.example to reflect new configuration structure.
Add ChatMessage Pydantic model for type-safe internal message handling: - role: ChatRole enum (USER, ASSISTANT, SYSTEM) - content: str - to_dict(): convert to dict for API compatibility Apply consistent multiline formatting to Qdrant config dicts.
Add format_beliefs_for_prompt_from_nodes() to format BeliefNode sequences for prompt construction without requiring async graph access. This enables request identity caching to pre-format beliefs once per turn. Output format matches existing format_beliefs_for_prompt() for consistency: - "topic: +0.50 (confidence: 0.80), evidence: 3" Apply consistent multiline formatting to function signatures and Cypher query parameters throughout MemoryGraph class.
Add token estimation and trimming utilities: - estimate_tokens_utf8(text): fast provider-agnostic approximation using UTF-8 byte length // 4 heuristic (~4 bytes per token for English) - estimate_messages_tokens(messages): sum estimates across message list - trim_chat_messages_for_budget(messages, max_message_tokens): drop oldest messages until within budget, keeping at least min_tail_messages - message_tokens_budget_for_system(total_budget, system_prompt, reserve): compute headroom for user/assistant turns after system prompt Used by agent to prevent input token overflow on long conversations.
Add ContextVar-based request-scoped identity caching: - IdentityBundle: frozen dataclass with snapshot_text, beliefs_prompt_text, and all_beliefs tuple (loaded once per user-facing request) - get_request_identity(): retrieve cached bundle for current context - set_request_identity(bundle): bind bundle, returns token for reset - reset_request_identity(token): restore previous context in finally block Eliminates redundant Neo4j reads when building context, reflecting, and assessing provenance within the same interaction turn.
Extract core identity and instructions into SYSTEM_PROMPT_STATIC_CACHED constant for prompt caching optimization. Dynamic per-request content (personality state, beliefs, memories, knowledge) appended separately. New structure: - SYSTEM_PROMPT_STATIC_CACHED: ## Core Identity + ## Instructions - build_system_prompt_dynamic(): returns session-specific sections only - build_system_prompt(): combines static prefix + dynamic content Prompt changes: - Add 150-word default response length limit - Clarify knowledge grounding: stored facts may be outdated, honor corrections - Simplify routing/reranking/decomposition/forgetting prompt prefixes: - "Classify query for memory retrieval" - "Decompose complex query into independent sub-queries" - "Rank candidates by relevance to query" - "Review memory candidates for archival/forgetting"
…ency provider.py: - Add strip_analysis_block() to remove <analysis>...</analysis> scratchpad blocks from LLM output before JSON extraction - Use config.LLM_CONCURRENCY for semaphore (default 1, raise for multi-GPU) - Integrate strip_analysis_block() into decode_llm_json() llm/caller.py: - Use config.LLM_MAX_RETRIES instead of hardcoded _MAX_RETRIES (3) - Use config.LLM_BACKOFF_BASE instead of hardcoded _BACKOFF_BASE (2.0) - Add decimal precision guidance to JSON system prompt
- Use config.ESS_MAX_TOKENS (512) instead of LLM_MAX_TOKENS - Remove ESS_TIMEOUT_SECONDS constant (now config.ESS_TIMEOUT) - Add belief_update_recommended, urgency to PROVIDER_JSON_ONLY_NOTE - Add decimal precision guidance to reduce verbose float serialization - Apply consistent multiline formatting to alias dicts and log statements
consolidation.py: - STRUCTURED_JSON_MAX_TOKENS for readiness checks - EXTRACTION_MAX_TOKENS for summary generation db.py: - Use config.NEO4J_MAX_POOL_SIZE and NEO4J_CONNECTION_TIMEOUT derivatives.py: - EXTRACTION_MAX_TOKENS for semantic chunking embedder.py: - Use config.EMBEDDING_CACHE_SIZE for LRU cache forgetting.py: - Add max_retries=1 for fail-fast behavior knowledge_extract.py: - STRUCTURED_JSON_MAX_TOKENS for window context summary - EXTRACTION_MAX_TOKENS for proposition extraction segmentation.py: - STRUCTURED_JSON_MAX_TOKENS for boundary detection semantic_features.py: - EXTRACTION_MAX_TOKENS for feature extraction - STRUCTURED_JSON_MAX_TOKENS for consolidation belief_provenance.py, dual_store.py: - Apply consistent multiline formatting
chain.py: - STRUCTURED_JSON_MAX_TOKENS for sufficiency checks - Add max_retries=1 reranker.py: - RERANK_MAX_TOKENS (512) for episode reranking - Add max_retries=1 router.py: - STRUCTURED_JSON_MAX_TOKENS for query routing - Add max_retries=1 split.py: - STRUCTURED_JSON_MAX_TOKENS for query decomposition - Add max_retries=1 Apply consistent multiline formatting across all modules.
Public API: - Add max_tokens and temperature params to respond() and respond_stream() - Default to config values when not specified Request identity caching: - Add _fetch_identity_bundle() to load snapshot/beliefs once - Use IdentityBundle via ContextVar across _build_context, _reflect, _assess - Eliminate duplicate Neo4j reads within single turn Token budget trimming: - Apply trim_chat_messages_for_budget() before chat completion - Cap input at CHAT_INPUT_TOKEN_BUDGET (28000 tokens) ingest() changes: - Wrap in try/finally to properly reset request identity context - Use cached identity bundle for provenance assessment Config integration: - REFLECTION_USER_SLICE, REFLECTION_AGENT_SLICE for message slicing - FORGETTING_CANDIDATE_LIMIT for forgetting cycle
Request tracing: - RequestIDMiddleware: add X-Request-ID header to all responses - Accept client-provided X-Request-ID or generate uuid4 - get_request_id() for logging context Authentication: - Optional Bearer token auth via SONALITY_HTTP_API_KEY env var - Skip auth for /health and /v1/health endpoints - Return 401 for missing/invalid key when configured Parameter passthrough: - Pass max_tokens (clamped to LLM_MAX_TOKENS) to agent - Pass temperature to agent - Apply to both streaming and non-streaming paths Health endpoint: - Add uptime_seconds (time since startup) - Add version field from __version__
Fix shared history bug where all Telegram users saw each other's context when using single shared SonalityClient. ClientPool class: - get(user_id): get or create client, update last access time - cleanup(): remove clients idle > idle_timeout (default 1 hour) - close_all(): graceful shutdown of all clients - Async-safe with asyncio.Lock protection Handler changes: - Replace client injection with pool injection - Handlers call pool.get(uid) to obtain per-user client - cmd_beliefs, cmd_health, handle_text, handle_voice updated Startup changes: - Create ClientPool at startup - Test connectivity with temporary client before polling
Apply multiline formatting to function calls and f-strings for improved readability in terminal.py, cli.py, and psych_harness.py.
Update mock prompt patterns to match new simplified prefixes: - test_reranker: "Rank candidates by relevance to query" - test_router: "Classify query for memory retrieval" - test_split: "Decompose complex query into independent sub-queries" - test_forgetting: "Review memory candidates for archival/forgetting" test_api.py: - Add max_tokens=2048 and temperature=0.3 to test request - Assert parameters passed to agent.respond() as kwargs test_chain.py: - Formatting only (collapse multiline to single line) test_forgetting.py: - Formatting only (consistent multiline for asyncio.run)
Configuration table: - Add task-specific token settings (ESS, STRUCTURED_JSON, EXTRACTION, RERANK) - Add Neo4j pool settings (MAX_POOL_SIZE, CONNECTION_TIMEOUT) - Add LLM retry settings (MAX_RETRIES, BACKOFF_BASE) - Add EMBEDDING_CACHE_SIZE, HTTP_API_KEY - Remove deprecated: REFLECTION_EVERY, OPINION_COOLING_PERIOD, BOOTSTRAP_DAMPENING_UNTIL, EMBEDDING_BASE_URL, EMBEDDING_MODEL, EMBEDDING_SEND_DIMENSIONS, SEMANTIC_RETRIEVAL_COUNT, EPISODIC_RETRIEVAL_COUNT Slash commands: - Keep: /snapshot, /beliefs, /models, /quit - Remove: /sponge, /insights, /staged, /topics, /shifts, /health, /diff, /reset Architecture: - ESS-based update gating description - Two-tier reflection (triage + optional deep) - Note vector retrieval (future BM25+RRF) - Update test count: 65/65 (excluding live tests)
Foundation for unified tool system: - ToolName: canonical names for all 7 agent tools - EventType: progress event types for UX streaming - AssessFocus: modes for evidence assessment tool These enums are imported throughout the new tool system.
Add Firecrawl integration config: - WEB_SEARCH_URL: Firecrawl API endpoint (default localhost:3002) - WEB_SEARCH_ENABLED: master toggle for web access - WEB_CACHE_TTL: search result cache duration (4h default) - REFLECTION_WEB_QUERIES: max queries per reflection enrichment
New function for tools that need plain text output rather than JSON parsing. Used by assess_evidence and consolidate tools. Complements existing llm_call() for structured responses.
Update CORE_IDENTITY to emphasize recall-verify-assess-reflect cycle. New prompts: - ASSESS_EVIDENCE_PROMPT: gap/contradiction/quality analysis - CONSOLIDATION_TOOL_PROMPT: research synthesis - REFLECTION_WEB_SECTION: web evidence context for reflection Updated system instructions for tool-driven cognition.
New module with: - AgentEvent dataclass: type, detail, tool_name, iteration, sources_count - Event type constants re-exported from schema - noop_progress callback for code paths without progress reporting Enables chat clients to show live status during agentic loop.
New sonality/web module: search.py: - WebSearchClient wrapping Firecrawl /v1/search and /v1/scrape - SearchResult/Response/ExtractResult dataclasses - Async with caching, rate limiting, multi_search context.py: - sanitize_web_content(): strips injection patterns, zero-width chars - format_web_context(): dual delimiter defense (XML + numbered) - Regex patterns for common injection attacks __init__.py: get_client() factory with config-based initialization
New sonality/tools module: __init__.py: - ToolContext: runtime dependencies decoupled from agent - dispatch_tool(): execute tool by name - get_definitions(): all tool schemas for LLM Tool modules (each exports DEFINITIONS + EXECUTORS): - memory.py: recall_memory, store_knowledge - web.py: web_search, web_extract - assess.py: assess_evidence (gaps/contradictions/quality/summary) - consolidate.py: mid-loop research synthesis - reflect.py: belief updates with web enrichment, run_forgetting LLM autonomously selects tools; no hardcoded cognitive stages.
Adapt memory architecture to work with ToolContext: - forgetting.py: assess_and_forget callable from tools/reflect.py - knowledge_extract.py: exposed via store_knowledge tool - graph.py: methods used by reflect tool for belief updates - retrieval/*: used by recall_memory tool No breaking changes to public interfaces.
- ess.py: classifier used in bookkeeping phase - provider.py: StreamChunk used for agentic streaming - token_budget.py: summarize_and_trim for context management Minor adjustments to support unified tool loop.
Replace hardcoded cognitive stages with LLM-driven tool selection. Key changes: - _run_agentic_loop: yields AgentEvent during reasoning, StreamChunk for response - _make_tool_context: bridges agent state to tools package - _dispatch_tools: execute tool calls with result recording - _dedup_tool_calls: prevents identical repeat calls The LLM decides workflow via tool calls: recall, search, assess, consolidate, reflect, store. Progress callbacks enable real-time UX. Hard ceiling (50 steps) prevents infinite loops.
api.py: - SSE streaming emits AgentEvent as typed events - Progress events sent before content chunks - Event types: thinking, tool_call, tool_result, context_build, done cli.py: - Imports updated for new progress module - Agent initialization unchanged
client.py: - ProgressEvent dataclass for parsed SSE events - TOOL_LABELS: human-readable tool names - pipeline_summary(): compact tool chain description terminal.py: - Live status panel during tool execution - _progress_action(): convert events to display strings telegram.py: - Typing indicators during tool execution - Progress events logged for debugging
Full Firecrawl deployment for web search and scraping: - firecrawl: API server on port 3002 - firecrawl-playwright: Playwright renderer for JS-heavy sites - firecrawl-redis: caching and rate limiting - firecrawl-rabbitmq: job queue - firecrawl-postgres: persistent storage - searxng: multi-engine search aggregation SearXNG config (config/searxng/settings.yml): - Engines: Google, DuckDuckGo, Bing, Wikipedia, GitHub, arXiv - JSON output format enabled - Rate limiting and ban time configured Resource limits prevent runaway containers.
.env.example: - Document SONALITY_WEB_SEARCH_URL, WEB_SEARCH_ENABLED, WEB_CACHE_TTL - Add web access section with defaults pyproject.toml: - Add httpx>=0.28.0 for async HTTP in WebSearchClient
BookkeepingItem queue payload. classify_ess() with topic normalization and existing graph topics context. enqueue_bookkeeping() thread-safe non-blocking put. process_bookkeeping() consumer: store episode then shared pipeline. post_ingest() synchronous ingest path. _run_pipeline: provenance -> semantic features -> knowledge extraction -> forgetting.
Replace SearchResult/ExtractResult with ResearchFact (claim + confidence + source metadata). Add ResearchProgress with SSE event mapping. Multi- line SSE data buffering with terminal events. Trace propagation via X-Trace-ID. Parse facts from API instead of document. Remove search()/ multi_search()/extract(). 5h read timeout.
Add ProgressCallback and _noop_progress. Extend ToolContext with research_transcript callable, short_term_memory, progress callback. Add ToolLabeler registry and tool_label() for UI/SSE display. Remove build_research_transcript() and synthesize from lazy loader. Import Embedder from shared.
Replace web_search and web_extract with unified web_research at all depth presets (glance through exhaustive). Add ResearchArgs and parse_research_args(). merge_facts() dedupes streamed vs final. format_facts() groups by source with confidence tags. Progress via ctx.progress(). STM injected into research goal. HTTP 429 handling.
Delete evidence synthesis tool. Research now returns structured facts — agent reasons over facts and calls integrate_knowledge.
Add ctx.progress() at recall/integrate stages. Deterministic episode identity via deterministic_id. Partial-store degradation messages. Add ctx.short_term_memory in recall output. Use ctx.research_transcript for reflect. Richer section headers with counts.
Replace LLM-based belief reranking with rank_beliefs_by_similarity (embedding cosine) and rank_beliefs_algorithmically (RRF fusion of embedding + graph confidence * log evidence). Single reasoning model LLM call. Web enrichment via start_research depth=glance. Remove consensus_call, _merge_reflections, forgetting cycle.
Use shared RequestIDMiddleware, HealthResponse, check_health_ dependencies. Constant-time bearer token auth via hmac.compare_digest. Stream wrapped in try/except yielding SSE error event. Max tokens upper bound 131072. Remove /chat endpoint. Ingest response: CredibilitySignals dict + float urgency. Pydantic range constraints on belief fields.
LoopState per-request instead of instance fields. THINKING/ACTING phase machine: _phase_thinking (LLM + compose_guarded, stall/extend handling), _phase_acting (dispatch + MemoryUpdate consolidation, raw tool output never re-enters THINKING). Async bookkeeping queue. _pre_seed_memory for episodic LTM retrieval. _flush_ltm_to_episode. _rank_beliefs with embedding cosine. Streaming Fathom research via thread-safe queue. check_dependencies health probe. Shutdown drain with 10s timeout. Remove context folding, message compression, loop handoff, boundary detection, segment consolidation.
Config: add tts_optimize_concurrency, pydantic validator for TTS fallback defaults, set tts_optimize_max_tokens to -1, default log level DEBUG. Package init: entry-point only, logging from settings.
Replace local llm_call wrapper with compose_guarded() + provider chat_completion. LLMProvider gets concurrency from settings. _TTS_SYSTEM as Final system prompt. Context char limit derived from max_tokens.
Per-request trace_id with X-Trace-ID header and structlog binding. Rename web_search/web_extract tool labels to web_research. Add goal to arg summary scan. Harden SSE parsing with safe empty choices. Fix empty assistant response history corruption. Add tool_progress logging.
Add tool_progress handling with intermediate status updates. Scrolling status window (last 8 steps, max 10 lines). Remove reviewing progress, ESS score/reasoning_type/topics from done footer. Structured elapsed_s.
Stdlib-only HuggingFace GGUF downloader. Default output to .models/. Models: llm (Qwen3.6-35B Q4_K_XL), embed (Qwen3-Embedding-4B Q4_K_M). Selective --all, --list, --force flags. Resumable downloads with .part files. Optional --verify SHA-256.
Feed: strip HTML tags from article description before MIN_DESC_LEN check. Diagnostics: migrate config access to pydantic settings API, remove neo4j_derivatives count and derivative density health rules.
Delete tests/containers.py (Qdrant/Neo4j testcontainers lifecycle). Remove --use-containers option, session db_containers fixture, autouse clear_db_between_tests, multi-target mock_llm_call fixture. Add minimal env defaults. Root conftest: drop benches scope, suppress_expected_logs.
Regression tests for clean_completion, decode_llm_json, extract_tool_calls, strip_markdown, message_content_text, normalize_llm_list_response, coerce_string_fields.
Fathom: remove URLScoring tests, split PageAnalysisResult tests for dict and bare list input. Derivatives: patch async_llm_call and async_embed_documents with AsyncMock. API: update _make_ess() for CredibilitySignals + float urgency, mock check_dependencies and http_dependency_status.
Update chat clients, scripts, and test suite
Rewrite Sonality agent: two-phase automaton, unified tools, async bookkeeping
Refactor Sonality memory: graph schema, ESS storage, unified retrieval
Refactor Sonality core: config, ESS signals, prompts, caller
Refactor Fathom: fact-centric research, embedding ranking, source memory
Feat/shared module refactor
Remove benchmark suite and rewrite documentation
Infrastructure: AGPL license, local llama.cpp stack, CI updates
Restructure into monorepo with shared infrastructure and fathom service
…wledge Structured agentic loop with pipeline enforcement and integrate_knowledge
feat: Unified agentic tool system with web search
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance and reliability improvements across 37 files (35 modified, 2 new).
Task-Specific Token Budgets
Request Identity Caching
IdentityBundle(ContextVar)Chat Input Trimming
trim_chat_messages_for_budget(): drop oldest messages when exceedingCHAT_INPUT_TOKEN_BUDGETAPI Enhancements
X-Request-IDheader for distributed tracingSONALITY_HTTP_API_KEYmax_tokensandtemperaturepassthrough to agentPrompt Caching Optimization
Provider Improvements
strip_analysis_block(): remove<analysis>scratchpad before JSON extractionLLM_CONCURRENCYconfig for multi-GPU deploymentsTelegram Client Isolation
ClientPool: per-user SonalityClient instances (fixes shared history bug)Commits (16)
Test Plan
make test— 65/65 unit tests passmake bench-contracts— 80/80 contracts passSONALITY_HTTP_API_KEY, verify 401 without Bearer tokenX-Request-IDheader in responses