Refactor Fathom: fact-centric research, embedding ranking, source memory#90
Merged
Okm165 merged 52 commits intoJun 10, 2026
Merged
Conversation
Inherit InfraSettings to eliminate duplicated infra fields. Add browser_concurrency, sample_temperature, qdrant_search_ef, exploration_divergence, session_timeout. Set llm_max_tokens=-1, llm_request_timeout=3600.
Module-level provider via settings.make_llm_provider(). Add async concurrency gates. Introduce async_llm_call, format_prompt, async_embed_query, async_embed_documents via make_gated_embedders.
Provider creation inlined into caller via InfraSettings factory. CLI entry point and Rich ResearchDashboard removed — Fathom is API-only. Delete __main__.py, cli.py, observe.py, provider.py.
Remove URLScoring, Contradiction, ResearchOutput, WebSearch/WebExtract types. Replace has_evidence boolean with source_quality float. Add DomainStats, extract_domain(), FactWithSource. Add depth presets (glance through exhaustive). Field validation with clamping.
Remove document from ResearchSession, add timed_out status. Add source_quality on ResearchFact. Batch add_urls/insert_facts/ save_checklist with UNWIND Cypher. Randomize pending URL order. Add SourceDomain/TopicCluster graph nodes. Add cleanup_stale_sessions and check() health probe.
Remove SCORE_URLS, UPDATE_CHECKLIST, COMPRESS_KNOWLEDGE, WRITE_SECTION, WRITE_INTRO_CONCLUSION prompts. Rewrite decomposition for search-query- friendly questions and page analysis for source_quality. LLM surface reduced to decompose_goal, generate_queries, analyze_page only.
Introduce rank_urls_hybrid() with RRF fusion of embedding similarity and session domain quality, softmax-temperature sampling. Add build_ranked_knowledge_context() for page analysis. Delete LLM-based selection.py (probabilistic_sample, format_urls_for_scoring).
Qdrant fathom_sources collection for persistent source intelligence. record_source() upserts embedding and updates SourceDomain/TopicCluster graph. suggest_sources() ranks by vector similarity weighted by domain quality. diverge_probabilistically() blends goal and known-URL centroid.
Browser: add check() health probe and fetch_preview_batch() for lightweight domcontentloaded previews. Extract: add extract_preview() with metadata/headings/lead text for quality signal before full fetch. Search: add 30s timeout on DuckDuckGo calls. Remove single-URL fetch().
Replace document synthesis with fact-gathering engine. Add 7 depth presets. Session timeout enforcement. Source memory integration. Embedding ranking pool with preview-then-select pipeline. HTML cache reuse. Ranked knowledge context for page analysis. Rich SSE via _emit. Remove stall detection, checklist answering, contradiction pipeline, document writing, dashboard parameter.
Add Qdrant lifecycle, stale session cleanup, RequestIDMiddleware, shared HealthResponse with Neo4j/Qdrant/browserless/LLM probes. Concurrency cap (2 sessions, HTTP 429). Per-session asyncio.Queue for real-time SSE. Return facts list instead of document. Remove /search and /extract endpoints. Clean __init__.py logging side effect.
Inherit shared InfraSettings to eliminate duplicated infra fields. Add ingest_temperature, max_stalls, max_extends, context_char_limit. Scale chat_input_token_budget to 150K, llm_timeout to 3600. Remove agent_compress_threshold/keep_tail, ess_timeout, web_search_enabled.
Module-level provider via settings.make_llm_provider(). Add async concurrency gates, format_prompt, async_llm_call, gated embedding entry points. Remove consensus_call. Delete provider.py (interaction gating, module-level default_provider).
Add normalize_topic() for canonical topic slugs. Add Phase enum for agent automaton. Add EventType.TOOL_PROGRESS. Replace QDRANT_COLLECTIONS constant with qdrant_collection_specs(dims) factory. Add prospective query and credibility signal floats to derivative payload. Remove Neo4j Derivative/Segment/Summary constraints. Reduce tool names to recall_memory, web_research, integrate_knowledge.
Introduce CredibilitySignals dataclass with five continuous 0-1 dimensions replacing ReasoningType, SourceReliability, OpinionDirection, KnowledgeDensity. Continuous float urgency replaces UrgencyLevel enum. Add existing_topics for dedup. Pydantic bounds on schema fields.
Add identity/confidentiality system instructions and cognitive discipline. Add CONSOLIDATION_INSTRUCTIONS for working-memory JSON replacing handoff/compression prompts. Rewrite ESS prompt for continuous credibility dimensions. Add WEB_EVIDENCE_HEADER, PROSPECTIVE_QUERY_PROMPT, build_ingest_system(). Remove LOOP_HANDOFF, STATE_COMPRESSION, BOUNDARY_DETECTION, SUFFICIENCY, DECOMPOSITION, CONSOLIDATION_READINESS/SUMMARY, SYNTHESIZE prompts. Rewrite belief prompts for bears_on_belief gate and signal weights.
Add _DEFAULT_COMPLETION_RESERVE (16K). Raise SUMMARIZE_THRESHOLD 8->20 and per-message slice 300->800 chars. Use format_prompt() and instructions= API. Add Field(max_length) on summary schema. Use render_labeled(). Remove multipart message content handling.
CLI: import ChatRole from shared.types, use structlog.get_logger. Progress: remove re-exported event constants (callers use EventType from schema directly).
Remove Derivative, Segment, Summary nodes and all related edges. Add five credibility signal fields to EpisodeNode. MERGE-based store_episode and link_belief with evidence count recomputation. FSRS power-law utility scoring. Rewire TEMPORAL_NEXT on delete. Add get_topic_names() and get_belief_connections(). Batch topic linking with normalize_topic.
Store: accept ESSResult with credibility signals. Qdrant Prefetch + FormulaQuery with per-signal additive boosts. Add prospective queries for ROXY-style vocabulary bridging. Add remove_knowledge_citations() for forgetting cascade. DB: accept optional embedding_dims, guard partial init on close.
Convert chunk_and_embed() to async using async_llm_call and async_embed_documents. Switch from local EmbedderProtocol to shared.embedder.Embedder. Add _MAX_CHUNKS=15 validation. Delete memory/embedder.py (FastEmbed wrapper consolidated into shared).
Introduce EpisodeEvidence dataclass bundling episode context, ESS score, and CredibilitySignals. Add _belief_snapshot() for prompt serialization. Create provenance edges only when bears_on_belief is true and evidence_strength > 0. Migrate to async_llm_call. Batch cap 30, normalize_topic.
Block FORGET when episode is sole evidence for a belief via get_belief_connections(). Call remove_knowledge_citations() on ARCHIVE and FORGET. Log but do not re-raise on Qdrant failure after graph delete. Async LLM with 50 decision validation cap.
Remove PropositionType enum, negation handling, type-based filtering, and confidence inflation. Citation accumulation only with MAX_CITATIONS cap. Async _split_windows and _extract_propositions. Use VECTOR_SEARCH_THRESHOLD. UID seed from full normalized text.
Delete EventBoundaryDetector, LLM boundary detection, segment counter persistence. Delete LLM-based segment readiness check and HEMA hierarchical summary generation. Graph Segment/Summary nodes already removed.
Router: add SearchPass with optional query rewrite and signal_weights. _sanitize_weights() clamps to ESS signal names (max 0.3, max 3 passes). Remove QueryCategory.MULTI_ENTITY and should_decompose. Async route_query(). Reranker: async with ESS/quality in episode lines. Overflow candidates appended after ranked portion.
New retrieve() single orchestrator: route -> multi-pass fetch (graph topic/belief hits + per-pass vector search with signal weights) -> temporal expand -> rerank -> access update -> optional semantic feature search -> formatted context strings. Delete chain.py (iterative sufficiency loop) and split.py (LLM decomposition). Export retrieve.
Rename SemanticIngestionWorker to SemanticFeatureExtractor. Replace background thread/queue/event loop with direct async process_episode(). Lazy consolidation at CONSOLIDATION_THRESHOLD=20. Injected AsyncQdrantClient. Per (tag, feature) dedup with citation merge and re-embedding.
Export EpisodeEvidence, SemanticFeatureExtractor, retrieve. Remove Embedder, StoredEpisode, BoundaryDecision, EventBoundaryDetector, SemanticIngestionWorker, chain_retrieve, split_retrieve.
Pure data models and stateless helpers: MemoryUpdate (LTM/STM consolidation schema), LoopStep, ActingContext, LoopState (per-request mutable state). build_step_context(), build_scaffolding(), synthesis_prompt(), dedup_tool_calls(), summarize_for_step_log(). No IO.
BookkeepingItem queue payload. classify_ess() with topic normalization and existing graph topics context. enqueue_bookkeeping() thread-safe non-blocking put. process_bookkeeping() consumer: store episode then shared pipeline. post_ingest() synchronous ingest path. _run_pipeline: provenance -> semantic features -> knowledge extraction -> forgetting.
Replace SearchResult/ExtractResult with ResearchFact (claim + confidence + source metadata). Add ResearchProgress with SSE event mapping. Multi- line SSE data buffering with terminal events. Trace propagation via X-Trace-ID. Parse facts from API instead of document. Remove search()/ multi_search()/extract(). 5h read timeout.
Add ProgressCallback and _noop_progress. Extend ToolContext with research_transcript callable, short_term_memory, progress callback. Add ToolLabeler registry and tool_label() for UI/SSE display. Remove build_research_transcript() and synthesize from lazy loader. Import Embedder from shared.
Replace web_search and web_extract with unified web_research at all depth presets (glance through exhaustive). Add ResearchArgs and parse_research_args(). merge_facts() dedupes streamed vs final. format_facts() groups by source with confidence tags. Progress via ctx.progress(). STM injected into research goal. HTTP 429 handling.
Delete evidence synthesis tool. Research now returns structured facts — agent reasons over facts and calls integrate_knowledge.
Add ctx.progress() at recall/integrate stages. Deterministic episode identity via deterministic_id. Partial-store degradation messages. Add ctx.short_term_memory in recall output. Use ctx.research_transcript for reflect. Richer section headers with counts.
Replace LLM-based belief reranking with rank_beliefs_by_similarity (embedding cosine) and rank_beliefs_algorithmically (RRF fusion of embedding + graph confidence * log evidence). Single reasoning model LLM call. Web enrichment via start_research depth=glance. Remove consensus_call, _merge_reflections, forgetting cycle.
Use shared RequestIDMiddleware, HealthResponse, check_health_ dependencies. Constant-time bearer token auth via hmac.compare_digest. Stream wrapped in try/except yielding SSE error event. Max tokens upper bound 131072. Remove /chat endpoint. Ingest response: CredibilitySignals dict + float urgency. Pydantic range constraints on belief fields.
LoopState per-request instead of instance fields. THINKING/ACTING phase machine: _phase_thinking (LLM + compose_guarded, stall/extend handling), _phase_acting (dispatch + MemoryUpdate consolidation, raw tool output never re-enters THINKING). Async bookkeeping queue. _pre_seed_memory for episodic LTM retrieval. _flush_ltm_to_episode. _rank_beliefs with embedding cosine. Streaming Fathom research via thread-safe queue. check_dependencies health probe. Shutdown drain with 10s timeout. Remove context folding, message compression, loop handoff, boundary detection, segment consolidation.
Config: add tts_optimize_concurrency, pydantic validator for TTS fallback defaults, set tts_optimize_max_tokens to -1, default log level DEBUG. Package init: entry-point only, logging from settings.
Replace local llm_call wrapper with compose_guarded() + provider chat_completion. LLMProvider gets concurrency from settings. _TTS_SYSTEM as Final system prompt. Context char limit derived from max_tokens.
Per-request trace_id with X-Trace-ID header and structlog binding. Rename web_search/web_extract tool labels to web_research. Add goal to arg summary scan. Harden SSE parsing with safe empty choices. Fix empty assistant response history corruption. Add tool_progress logging.
Add tool_progress handling with intermediate status updates. Scrolling status window (last 8 steps, max 10 lines). Remove reviewing progress, ESS score/reasoning_type/topics from done footer. Structured elapsed_s.
Stdlib-only HuggingFace GGUF downloader. Default output to .models/. Models: llm (Qwen3.6-35B Q4_K_XL), embed (Qwen3-Embedding-4B Q4_K_M). Selective --all, --list, --force flags. Resumable downloads with .part files. Optional --verify SHA-256.
Feed: strip HTML tags from article description before MIN_DESC_LEN check. Diagnostics: migrate config access to pydantic settings API, remove neo4j_derivatives count and derivative density health rules.
Delete tests/containers.py (Qdrant/Neo4j testcontainers lifecycle). Remove --use-containers option, session db_containers fixture, autouse clear_db_between_tests, multi-target mock_llm_call fixture. Add minimal env defaults. Root conftest: drop benches scope, suppress_expected_logs.
Regression tests for clean_completion, decode_llm_json, extract_tool_calls, strip_markdown, message_content_text, normalize_llm_list_response, coerce_string_fields.
Fathom: remove URLScoring tests, split PageAnalysisResult tests for dict and bare list input. Derivatives: patch async_llm_call and async_embed_documents with AsyncMock. API: update _make_ess() for CredibilitySignals + float urgency, mock check_dependencies and http_dependency_status.
Update chat clients, scripts, and test suite
Rewrite Sonality agent: two-phase automaton, unified tools, async bookkeeping
Refactor Sonality memory: graph schema, ESS storage, unified retrieval
Refactor Sonality core: config, ESS signals, prompts, caller
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan