Restructure into monorepo with shared infrastructure and fathom service#87
Merged
Okm165 merged 88 commits intoJun 10, 2026
Merged
Conversation
Extract common utilities into src/shared/: LLM transport (caller, provider, parse), structured logging with structlog, pydantic-settings config base, Neo4j connection helper, FastAPI server factory, common types, and error hierarchy.
Introduce src/fathom/: a standalone research agent with DuckDuckGo search, Playwright page fetching via browserless CDP, LLM-driven fact extraction and source grading, multi-step research sessions with SSE progress streaming. Replaces the firecrawl web scraping stack.
Move the core personality agent to src/sonality/ for the monorepo layout. Refactor to use src/shared/ for LLM transport, config, logging, and Neo4j. Replace inline web search with ResearchClient delegating to fathom. Simplify ESS parser and switch to pydantic-settings.
Move chat clients to src/chat/. Add pydantic-settings config, switch to structlog, update imports to use src/shared/ infrastructure.
Update pyproject.toml with src/ package paths, add fathom/shared/chat packages, new dependencies (ddgs, playwright, pydantic-settings, rich, structlog, trafilatura), and fathom entry points. Point Dockerfile to docker/sonality.Dockerfile. Update zensical.toml paths.
Remove firecrawl, playwright-service, redis, rabbitmq, postgres, and searxng containers. Add browserless (Playwright CDP) and fathom (web research API) services. Delete searxng configuration.
Add fathom, fathom-serve, docker-build-fathom targets. Update lint, format, typecheck paths to src/. Update .env.example with FATHOM_ config section, replace SONALITY_WEB_SEARCH_URL with SONALITY_FATHOM_URL.
Move sonality docs under docs/sonality/, add docs/fathom/ with overview and configuration guides. Update project-structure.md for monorepo layout.
Add tests/fathom/ and tests/sonality/ matching the src/ layout. Add fathom model tests and sonality API tests. Remove obsolete fixtures and tests superseded by integration testing. Update conftest loggers.
Network-server copyleft ensures modified versions offered over a network must provide corresponding source (Section 13).
Add AGPL-3.0-or-later metadata. Remove fastembed and testcontainers runtime deps. Add docs dependency group. Remove fathom CLI script entry point. Drop benches from ruff/pytest scope.
Local GGUF model weights directory should not be tracked.
Drop FASTEMBED_CACHE_PATH and ONNX model bake-in. Embeddings now served by llama.cpp sidecar via HTTP. Add UV_NO_PROGRESS and RUST_LOG=warn environment defaults.
llama-cpp (port 8080): ROCm GPU, Qwen3.6-35B Q4_K_XL, 262K context, flash attention, unified KV cache, jinja templates. llama-cpp-embed (port 8090): CPU-only, Qwen3-Embedding-4B Q4_K_M, pooling last, parallel 4. Wire Sonality and Fathom to both services. Bump browserless timeout 10x.
Switch defaults from OpenAI cloud to local llama.cpp. Document LLM_MODEL and EMBEDDING_MODEL GGUF stems, 262K context calibration, Fathom active defaults, TTS concurrency knob. Remove FastEmbed-specific vars and old compression settings.
Scope lint/format to src/ tests/. Switch type checker from mypy to pyright across all source packages. Remove bench contract test step. Add docs build verification with --group docs install.
Only trigger docs deploy on changes to docs/**, zensical.toml, or README.md. Add explicit Python 3.12 setup before uv.
Delete teaching harness (10K lines), scenario runner, all scenario files, and supporting conftest. Benchmark infrastructure is being rebuilt separately from the core agent.
Delete all bench test runners: ESS calibration, harness metrics, integrated live, knowledge accumulation, reflection cleanup, scenario contracts, and teaching suite tests.
Delete 29 bench variables and 38 bench-* targets. Scope lint/format to src/ tests/ only. Update check target to drop bench-contracts. Fix preflight-live-probe import to caller module. Update docs targets for dependency group. Fix reset target API call.
Drop benches from documented packages, file/console logger scope, and handler cleanup. Remove suppress_expected_logs fixture.
Delete old fathom/, sonality/, getting-started, and project-structure docs. Superseded by restructured documentation covering the unified multi-agent architecture.
Architecture (sonality, fathom, memory, shared), concepts (ESS, sponge, belief revision, retrieval), design decisions (agentic loop, sycophancy resistance, rejected approaches), deployment (quickstart, config, docker), development (workflow, testing). Rewrite docs index as unified landing page.
Rename site to Sonality, update URL, switch palette to indigo/cyan, restructure navigation. Rewrite README for multi-agent architecture, three quick start modes, technical contributions, research foundations.
Remove LLMValidationError subclass — validation failures stay as Pydantic ValidationError with LLMErrorCategory.VALIDATION. Clarify docstring for exception chain.
Introduce RRF_K constant, rrf_score() for reciprocal rank fusion, and scores_to_ranks() for converting score dicts to rank dicts. Used by sonality reflect and fathom ranking modules.
HTTP embedding client for llama.cpp /v1/embeddings: Qwen3 asymmetric query/document prefixes, MRL truncation, batching with retries, sync+async bridges, probe_embedding_dims() for runtime dimension detection, make_gated_embedders() factory with semaphore concurrency, cosine_similarity utility.
Add InfraSettings BaseSettings with LLM, Neo4j, Qdrant, and embedding fields, validators for bounds clamping, and make_embedder()/ make_llm_provider() factories. Add VECTOR_SEARCH_THRESHOLD shared constant. Broaden quiet_third_party_loggers() suppression. Remove package __version__ constant.
Add clean_completion() as single entry for thinking/reasoning recovery. Introduce render_labeled() for display rendering. Harden strip_markdown with regex leak stripping. Change JSON extraction to last-valid-object strategy. Privatize helpers behind clean_completion.
Delegate response cleanup to clean_completion(). Add finish_reason to ChatResult for truncation detection. Remove supports_thinking flag, chat_template_kwargs, forced min-1 semaphore, and verbose logging.
Rename SemanticIngestionWorker to SemanticFeatureExtractor. Replace background thread/queue/event loop with direct async process_episode(). Lazy consolidation at CONSOLIDATION_THRESHOLD=20. Injected AsyncQdrantClient. Per (tag, feature) dedup with citation merge and re-embedding.
Export EpisodeEvidence, SemanticFeatureExtractor, retrieve. Remove Embedder, StoredEpisode, BoundaryDecision, EventBoundaryDetector, SemanticIngestionWorker, chain_retrieve, split_retrieve.
Pure data models and stateless helpers: MemoryUpdate (LTM/STM consolidation schema), LoopStep, ActingContext, LoopState (per-request mutable state). build_step_context(), build_scaffolding(), synthesis_prompt(), dedup_tool_calls(), summarize_for_step_log(). No IO.
BookkeepingItem queue payload. classify_ess() with topic normalization and existing graph topics context. enqueue_bookkeeping() thread-safe non-blocking put. process_bookkeeping() consumer: store episode then shared pipeline. post_ingest() synchronous ingest path. _run_pipeline: provenance -> semantic features -> knowledge extraction -> forgetting.
Replace SearchResult/ExtractResult with ResearchFact (claim + confidence + source metadata). Add ResearchProgress with SSE event mapping. Multi- line SSE data buffering with terminal events. Trace propagation via X-Trace-ID. Parse facts from API instead of document. Remove search()/ multi_search()/extract(). 5h read timeout.
Add ProgressCallback and _noop_progress. Extend ToolContext with research_transcript callable, short_term_memory, progress callback. Add ToolLabeler registry and tool_label() for UI/SSE display. Remove build_research_transcript() and synthesize from lazy loader. Import Embedder from shared.
Replace web_search and web_extract with unified web_research at all depth presets (glance through exhaustive). Add ResearchArgs and parse_research_args(). merge_facts() dedupes streamed vs final. format_facts() groups by source with confidence tags. Progress via ctx.progress(). STM injected into research goal. HTTP 429 handling.
Delete evidence synthesis tool. Research now returns structured facts — agent reasons over facts and calls integrate_knowledge.
Add ctx.progress() at recall/integrate stages. Deterministic episode identity via deterministic_id. Partial-store degradation messages. Add ctx.short_term_memory in recall output. Use ctx.research_transcript for reflect. Richer section headers with counts.
Replace LLM-based belief reranking with rank_beliefs_by_similarity (embedding cosine) and rank_beliefs_algorithmically (RRF fusion of embedding + graph confidence * log evidence). Single reasoning model LLM call. Web enrichment via start_research depth=glance. Remove consensus_call, _merge_reflections, forgetting cycle.
Use shared RequestIDMiddleware, HealthResponse, check_health_ dependencies. Constant-time bearer token auth via hmac.compare_digest. Stream wrapped in try/except yielding SSE error event. Max tokens upper bound 131072. Remove /chat endpoint. Ingest response: CredibilitySignals dict + float urgency. Pydantic range constraints on belief fields.
LoopState per-request instead of instance fields. THINKING/ACTING phase machine: _phase_thinking (LLM + compose_guarded, stall/extend handling), _phase_acting (dispatch + MemoryUpdate consolidation, raw tool output never re-enters THINKING). Async bookkeeping queue. _pre_seed_memory for episodic LTM retrieval. _flush_ltm_to_episode. _rank_beliefs with embedding cosine. Streaming Fathom research via thread-safe queue. check_dependencies health probe. Shutdown drain with 10s timeout. Remove context folding, message compression, loop handoff, boundary detection, segment consolidation.
Config: add tts_optimize_concurrency, pydantic validator for TTS fallback defaults, set tts_optimize_max_tokens to -1, default log level DEBUG. Package init: entry-point only, logging from settings.
Replace local llm_call wrapper with compose_guarded() + provider chat_completion. LLMProvider gets concurrency from settings. _TTS_SYSTEM as Final system prompt. Context char limit derived from max_tokens.
Per-request trace_id with X-Trace-ID header and structlog binding. Rename web_search/web_extract tool labels to web_research. Add goal to arg summary scan. Harden SSE parsing with safe empty choices. Fix empty assistant response history corruption. Add tool_progress logging.
Add tool_progress handling with intermediate status updates. Scrolling status window (last 8 steps, max 10 lines). Remove reviewing progress, ESS score/reasoning_type/topics from done footer. Structured elapsed_s.
Stdlib-only HuggingFace GGUF downloader. Default output to .models/. Models: llm (Qwen3.6-35B Q4_K_XL), embed (Qwen3-Embedding-4B Q4_K_M). Selective --all, --list, --force flags. Resumable downloads with .part files. Optional --verify SHA-256.
Feed: strip HTML tags from article description before MIN_DESC_LEN check. Diagnostics: migrate config access to pydantic settings API, remove neo4j_derivatives count and derivative density health rules.
Delete tests/containers.py (Qdrant/Neo4j testcontainers lifecycle). Remove --use-containers option, session db_containers fixture, autouse clear_db_between_tests, multi-target mock_llm_call fixture. Add minimal env defaults. Root conftest: drop benches scope, suppress_expected_logs.
Regression tests for clean_completion, decode_llm_json, extract_tool_calls, strip_markdown, message_content_text, normalize_llm_list_response, coerce_string_fields.
Fathom: remove URLScoring tests, split PageAnalysisResult tests for dict and bare list input. Derivatives: patch async_llm_call and async_embed_documents with AsyncMock. API: update _make_ess() for CredibilitySignals + float urgency, mock check_dependencies and http_dependency_status.
Update chat clients, scripts, and test suite
Rewrite Sonality agent: two-phase automaton, unified tools, async bookkeeping
Refactor Sonality memory: graph schema, ESS storage, unified retrieval
Refactor Sonality core: config, ESS signals, prompts, caller
Refactor Fathom: fact-centric research, embedding ranking, source memory
Feat/shared module refactor
Remove benchmark suite and rewrite documentation
Infrastructure: AGPL license, local llama.cpp stack, CI updates
Okm165
merged commit Jun 10, 2026
3b0a0fd
into
feat/structured-loop-integrate-knowledge
1 check failed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/shared/): Extract common LLM transport, structured logging (structlog), pydantic-settings config, Neo4j helpers, FastAPI server factory, error hierarchy, and types into a reusable packagesrc/fathom/): New autonomous research agent replacing the firecrawl stack — DuckDuckGo search, Playwright page fetching via browserless CDP, LLM-driven fact extraction, source grading, multi-step sessions with SSE streamingsonality/→src/sonality/,chat/→src/chat/, docs intodocs/sonality/anddocs/fathom/Test plan
make install-devsucceeds with new package layoutmake lintandmake typecheckpass against src/ pathspytest tests/fathom/— fathom model tests passpytest tests/sonality/— sonality API tests passdocker compose upbuilds and starts all services (neo4j, browserless, fathom, sonality)/researchwith SSE stream