Knowledge Base — Manufacturing Search Engine

A precision information-retrieval service for manufacturing knowledge. Built on Elasticsearch with hybrid BM25 + vector search — not RAG, not generative AI. Documents are returned verbatim or not at all.

Why not RAG? In manufacturing, alarm codes differ by one character, equipment parameters are meaningless without domain context, and wrong answers have real consequences. This system is designed around a zero-fabrication guarantee: if a document matches, it is shown as-is; if nothing matches, the caller is told so explicitly.

Documentation Site
Architecture Overview
Prerequisites
Quick Start
- Option A — Docker Compose (recommended)
- Option B — Local (host Python + ES container)
Data Source — CSV Files
Configuration
AI Chat API
Taxonomy
Document Types
API Reference
Search Behaviour
File Import Pipeline
Development
Project Structure

Documentation Site

In-depth, bilingual (English / 中文) documentation lives in docs/ and is published as an interactive MkDocs Material site — architecture deep-dives, the search/ranking contract, the import pipeline, full configuration reference, API reference, and observability.

uv sync --extra docs           # install mkdocs-material + the i18n plugin
uv run mkdocs serve            # live preview at http://127.0.0.1:8000
uv run mkdocs build --strict   # build static site into ./site (fails on broken links)
uv run mkdocs gh-deploy        # publish to the gh-pages branch (GitHub Pages)

Auto-publish on push (no CI minutes)

Instead of GitHub Actions, the site is published by a local git hook. Run the installer once per clone:

./scripts/install-hooks.sh      # points core.hooksPath at scripts/git-hooks/

After that, any git push to main that touches docs/ or mkdocs.yml automatically builds (--strict) and deploys to gh-pages via scripts/deploy-docs.sh — no GitHub Actions, no billable minutes. A failed build aborts the push; bypass a single push with git push --no-verify. The installer also carries the repo's Git LFS hooks, so LFS keeps working. You can deploy by hand any time with ./scripts/deploy-docs.sh.

Use the language switcher in the site header to toggle English ↔ 中文. The pages below are a condensed quick-reference; the docs site is the canonical, deeper reference.

Architecture Overview

┌─────────────────────────────────────────┐
│          Upstream Chat Layer            │  ← extracts structured params from NL
└────────────────┬────────────────────────┘
                 │ SearchRequest (structured)
                 ▼
┌─────────────────────────────────────────┐
│         FastAPI (kb.main)               │
│                                         │
│  POST /api/v1/search                    │
│  POST /api/v1/documents/{type}          │
│  GET  /api/v1/facets                    │
└──────┬──────────────┬───────────────────┘
       │              │
       ▼              ▼
┌──────────┐   ┌───────────────────────────┐
│  ES 8.x  │   │  DashScope Embeddings API │
│ (BM25 +  │   │  text-embedding-v3        │
│  kNN)    │   │  (1024-dim, OpenAI-compat)│
└──────────┘   └───────────────────────────┘

Retrieval strategy: structured filters narrow the candidate set first, then hybrid BM25 keyword search + dense vector similarity re-ranks results using Reciprocal Rank Fusion (RRF). The caller never sees AI-generated text — only verbatim document sections.

Prerequisites

There are two ways to run the project — pick the row that matches your goal:

Goal	What you need
Just run it (deploy / try it out)	Docker + Docker Compose 24+ — see Option A. Nothing else.
Develop / modify code	Python 3.12+, uv, and Docker (for Elasticsearch) — see Option B.

An LLM API key (KB_LLM__API_KEY) and an embedding key (KB_EMBEDDING__API_KEY) are optional — the server boots without them and degrades gracefully (keyword-only search, AI chat disabled). See Configuration.

Quick Start

The whole stack — the API plus an Elasticsearch with the IK Chinese analyzer — is containerized. Option A is the fastest way to get running and the recommended path for deploying on any machine. Option B runs the app from source for development.

Option A — Docker Compose (recommended)

Everything runs in containers. The only requirement is Docker.

# 1. Clone, then (optionally) add your API keys
cp .env.example .env          # edit .env to set KB_LLM__API_KEY / KB_EMBEDDING__API_KEY
                              # (skip this and the app still runs — keyword-only, AI chat off)

# 2. Build and start the whole stack (ES + IK plugin + API)
docker compose up -d --build  # first build ~2-3 min; subsequent starts are instant

That's it. Open http://localhost:8000.

URL	Description
`http://localhost:8000`	Knowledge Base Search UI
`http://localhost:8000/docs`	Swagger UI / interactive API docs
`http://localhost:8000/redoc`	ReDoc API reference
`http://localhost:9200`	Elasticsearch (direct)

Everyday commands:

docker compose logs -f app        # follow API logs
docker compose ps                 # service status + health
docker compose restart app        # restart the API (e.g. after editing config/*.csv)
docker compose down               # stop everything (keeps ES data + uploads)
docker compose down -v            # stop and wipe the Elasticsearch data volume

What the compose stack gives you:

The app service waits for Elasticsearch to be healthy before starting, and reaches it over the internal network (KB_ES__URL=http://elasticsearch:9200 is set automatically — no need to configure it).
Persistence: ES data lives in the es-data named volume; uploaded/imported files in ./data/uploads; your CSVs and taxonomy.yaml are bind-mounted from ./config, so edits on the host take effect on the next docker compose restart app.
API keys are read from .env (optional). Anything in .env is passed through to the container.

Enabling OCR (scanned PDFs/images): OCR (PaddleOCR) is left out of the default image to keep it slim (~440 MB). To bake it in, set the build arg in docker-compose.yml:

# docker-compose.yml → services.app.build.args
INSTALL_OCR: "true"      # adds ~1.5-2 GB; models download on first use

then rebuild: docker compose build app && docker compose up -d.

Without the IK plugin / a different analyzer: the bundled ES image installs IK automatically. If you point the app at an external cluster that lacks IK, set KB_ES__ANALYZER_INDEX=cjk and KB_ES__ANALYZER_QUERY=cjk in .env.

Option B — Local (host Python + ES container)

Run the app from source for development; Elasticsearch still runs in Docker.

1. Install dependencies

uv sync --extra ingest          # app + file-import libs
# or for the full dev setup (tests, linters):
uv sync --extra dev --extra ingest
# plain pip equivalent: pip install -e ".[ingest]"

2. Start Elasticsearch only

docker compose up -d --build elasticsearch   # builds the IK image (one-time, ~1 min)
curl -s http://localhost:9200/_cluster/health | python3 -m json.tool
# "status": "green" or "yellow" means ready

3. Configure (optional)

cp .env.example .env   # set KB_LLM__API_KEY / KB_EMBEDDING__API_KEY if you want AI/vector features

.env is loaded automatically by pydantic-settings — no source needed. Shell exports always win over .env.

4. Start the API server

uv run python -m kb --reload

The port defaults to 8000; override it (highest priority wins): --port 8001 flag → KB_SERVER__PORT=8001 inline → .env. See Development for running multiple instances and the dev workflow.

What happens on startup

On every start (either option) the server automatically:

Creates Elasticsearch indices (kb_alarm, kb_setup, kb_experience, kb_import_files) if they don't exist
Clears and re-seeds all documents from the CSV files in config/ into ES
Restores previously imported documents from the kb_import_files tracker index
Serves the frontend at http://localhost:<port>

Why IK? The built-in cjk analyzer does CJK bigram tokenization and works without any plugin. IK (ik_max_word / ik_smart) uses a dictionary-based tokenizer that produces better recall for Chinese manufacturing terms. The elasticsearch/Dockerfile installs it automatically — no manual exec step needed.

Embedding service (optional): The server calls the DashScope Embeddings API (text-embedding-v3) for vector search. Set KB_EMBEDDING__API_KEY to enable it. Without a key the server runs in BM25-only mode — keyword search works fully, kNN semantic search is disabled.

Troubleshooting startup

embedding service unavailable warning at startup

If KB_EMBEDDING__API_KEY is not set or the DashScope API is unreachable you will see a log line like:

WARNING  seed: embedding service unavailable — docs indexed without vectors.
Keyword (BM25) search is fully available. Vector-only (kNN) fallback is disabled.
To enable: set KB_EMBEDDING__API_KEY and restart the server.

Keyword search works in full without a key. The only feature that requires stored vectors is the pure vector_only kNN fallback (the last step in the auto pipeline). BM25+vector re-scoring at query time also requires the API to be reachable.

To enable full kNN support: set KB_EMBEDDING__API_KEY in .env then restart uvicorn.

Cannot connect to Elasticsearch

Docker Compose (Option A): the app service reaches ES at http://elasticsearch:9200 (set automatically) and waits for its healthcheck. Check docker compose ps — if elasticsearch is not healthy, inspect docker compose logs elasticsearch. Two common causes: it needs more memory (the image requests 1 GB heap), or on stricter Linux hosts the kernel's vm.max_map_count is too low — raise it with sudo sysctl -w vm.max_map_count=262144 (persist in /etc/sysctl.conf).
Local (Option B): the app connects to http://localhost:9200 from config/settings.yaml. Confirm the container is up: docker compose ps elasticsearch.

The ES container in docker-compose.yml runs with xpack.security.enabled=false (plain HTTP, no auth) — suitable for local/single-node use.

Data Source — CSV Files

The system loads its knowledge base from three CSV files in config/. These are the authoritative corporate data source; the server reads and indexes them automatically on startup (only when the Elasticsearch index is empty).

File overview

File	Type	ES index	Rows (current)
`机台报警_header.csv`	Machine alarms	`kb_alarm`	100
`机台setup_header.csv`	Equipment setup / calibration	`kb_setup`	100
`设备经验_header.csv`	Field experience / failure cases	`kb_experience`	100

Column mapping

机台报警_header.csv → alarm documents

CSV column	ES field	Notes
项目	`project`	Must match a value in `taxonomy.yaml`
机台	`equipment`	Must match a value in `taxonomy.yaml`
代码	`error_codes`	Numeric or alphanumeric (e.g. `120001`, `SP-042`)
中文标题	`title` (prefix)	Combined with 英文标题 as `"中文（英文）"`
英文标题	`title` (suffix)
内容	`content`	Alarm description
解除流程	`resolution`	Step-by-step resolution
注意事项	`notes`	Warnings
ppt文件	`source_file`	Source document filename
ppt页面	`source_pages`	Page number(s)

机台setup_header.csv → setup documents

CSV column	ES field	Notes
项目	`project`
设备	`equipment`
工站/部件/站位	`title`	Auto-generated: `"{设备} · {station} 调试"`
规格/要求	`prerequisites`	First line
调试工具	`prerequisites`	Second line (appended)
调试步骤	`procedure`	Setup steps
注意事项	`notes`
ppt文件	`source_file`
PPT页面	`source_pages`

设备经验_header.csv → experience documents

CSV column	ES field	Notes
项目	`project`
机台	`equipment`
问题	`title`
失败描述	`body_text`	Opening paragraph
失败分析	`body_text`	Appended as `【失败分析】…`
根因	`body_text`	Appended as `【根因】…`
纠正步骤	`procedure`	Corrective actions
PPT文件	`source_file`
PPT页面	`source_pages`

How to update the knowledge base

Option A — Edit CSV files (traditional):

Edit one or more of the three CSV files (keep the header row unchanged).
Add any new projects or equipment names to config/taxonomy.yaml and reload:
```
curl -X POST http://localhost:8000/api/v1/admin/reload-taxonomy
```

Delete the affected ES index (the server will re-seed on next restart):

# Find the concrete index name behind the alias
curl http://localhost:9200/_alias/kb_alarm
# Delete it (replace kb_alarm_v1 with the actual name)
curl -X DELETE http://localhost:9200/kb_alarm_v1

Restart the server:
```
uv run python -m kb --reload
```

Option B — Import files (recommended for production):

Navigate to the 导入 Import page in the web UI, upload PDF/XLSX/PPTX/DOCX/CSV files (or scan a server folder), review the LLM-extracted documents, and commit. See File Import Pipeline for details.

Duplicate rows: rows that produce identical content hash (same title + content + project + equipment) are deduplicated automatically — only one copy is stored in ES.

Missing CSV: if a CSV file is absent, the server logs a warning and skips that document type.

Configuration

Settings are loaded in priority order: config/settings.yaml → .env (auto-loaded, git-ignored) → shell environment variables. Use .env.example as a starting template.

# config/settings.yaml
es:
  url: "http://localhost:9200"   # plain HTTP; no auth for local dev
  index_prefix: "kb"
  request_timeout_s: 10
  verify_certs: false
  analyzer_index: "ik_max_word"  # IK plugin (installed via elasticsearch/Dockerfile)
  analyzer_query: "ik_smart"     # fallback: set both to "cjk" if IK is not installed
  # For production with TLS + auth, uncomment:
  # url: "https://my-cluster:9200"
  # username: "elastic"
  # password: "..."             # or use KB_ES__PASSWORD env var
  # ssl_fingerprint: "dfbe360e..."

embedding:
  url: "https://dashscope.aliyuncs.com/compatible-mode/v1"  # OpenAI-compatible endpoint
  model: "text-embedding-v3"    # DashScope 1024-dim model
  dims: 1024
  batch_size: 32
  timeout_s: 30
  # api_key: ""                 # set via KB_EMBEDDING__API_KEY — never commit a real key

search:
  strict_max_hits: 8             # results above this → TOO_MANY, not shown
  title_boost: 3.0               # title field weight vs body (BM25)
  rrf_window: 50
  rrf_rank_constant: 60

taxonomy:
  path: "config/taxonomy.yaml"

server:
  host: "0.0.0.0"
  port: 8000   # override with KB_SERVER__PORT to run multiple instances

llm:
  api_url: "https://api.deepseek.com/v1/chat/completions"   # default: DeepSeek
  model: "deepseek-chat"
  max_tokens: 1200
  api_key: ""   # leave empty here — set KB_LLM__API_KEY in the environment instead

Common env-var overrides

Environment variables use the KB_ prefix and __ as the nesting delimiter:

KB_ES__URL=https://my-cluster:9200
KB_ES__PASSWORD=secret
KB_ES__SSL_FINGERPRINT=dfbe360e...    # SHA-256 of the server TLS cert
KB_ES__ANALYZER_INDEX=cjk             # fallback if IK plugin is not installed
KB_ES__ANALYZER_QUERY=cjk            # fallback if IK plugin is not installed
KB_EMBEDDING__API_KEY=sk-...          # DashScope key — required for vector search
KB_EMBEDDING__URL=https://dashscope.aliyuncs.com/compatible-mode/v1
KB_EMBEDDING__MODEL=text-embedding-v3
KB_EMBEDDING__DIMS=1024
KB_LLM__API_KEY=sk-...                # required to enable AI chat features
KB_LLM__API_URL=https://api.openai.com/v1/chat/completions   # switch LLM provider
KB_LLM__MODEL=gpt-4o-mini
KB_LLM__MAX_TOKENS=1200
KB_SERVER__PORT=8001                  # run on a non-default port
KB_SERVER__HOST=0.0.0.0              # bind address

TLS fingerprint (production)

To get the fingerprint of your Elasticsearch TLS certificate:

openssl s_client -connect localhost:9200 -showcerts 2>/dev/null \
  | openssl x509 -fingerprint -sha256 -noout

AI Chat API

The server includes a LLM proxy layer that keeps API keys server-side and away from the browser. Two endpoints are exposed:

Endpoint	Purpose
`POST /api/v1/chat`	Forward a conversation to the configured LLM
`POST /api/v1/extract`	Extract structured search parameters from a free-text query using the LLM, primed with the live taxonomy

Default provider — DeepSeek

Out of the box the server points at DeepSeek (deepseek-chat), which implements the OpenAI Chat Completions API wire format:

# config/settings.yaml
llm:
  api_url: "https://api.deepseek.com/v1/chat/completions"
  model: "deepseek-chat"
  max_tokens: 1200
  api_key: ""   # set via KB_LLM__API_KEY — never commit a real key

Get a key at platform.deepseek.com.

Setting your API key

Copy .env.example to .env (git-ignored) and set your key:

cp .env.example .env
# then edit .env — only KB_LLM__API_KEY is required

.env is loaded automatically by pydantic-settings on server startup — no source or wrapper command needed:

uv run python -m kb --reload   # .env is read automatically

You can still override any variable inline or via the shell:

# One-off inline override (takes precedence over .env)
KB_LLM__API_KEY=sk-... uv run python -m kb --reload

pydantic-settings uses env_prefix="KB_" and env_nested_delimiter="__". Shell exports always win over .env values.

Switching to a different AI provider

Any provider that implements the OpenAI Chat Completions API (POST /v1/chat/completions) works without code changes. Override the URL and model via environment variables:

Provider	`KB_LLM__API_URL`	`KB_LLM__MODEL`
DeepSeek (default)	`https://api.deepseek.com/v1/chat/completions`	`deepseek-chat`
OpenAI	`https://api.openai.com/v1/chat/completions`	`gpt-4o-mini`
Azure OpenAI	`https://<resource>.openai.azure.com/openai/deployments/<deployment>/chat/completions?api-version=2024-08-01-preview`	(set by deployment)
Ollama (local)	`http://localhost:11434/v1/chat/completions`	`qwen2.5:7b`
Any OpenAI-compat	your endpoint	your model name

Example — switch to OpenAI gpt-4o-mini:

export KB_LLM__API_KEY=sk-your-openai-key
export KB_LLM__API_URL=https://api.openai.com/v1/chat/completions
export KB_LLM__MODEL=gpt-4o-mini
uv run python -m kb --reload

All LLM environment variables

Variable	Default	Description
`KB_LLM__API_KEY`	(empty)	API key for the provider. Required to enable AI chat features.
`KB_LLM__API_URL`	`https://api.deepseek.com/v1/chat/completions`	Chat completions endpoint URL.
`KB_LLM__MODEL`	`deepseek-chat`	Model name passed to the provider in the request body.
`KB_LLM__MAX_TOKENS`	`1200`	Maximum tokens in the LLM response.

Behaviour when no API key is configured

If KB_LLM__API_KEY is not set:

POST /api/v1/chat returns HTTP 503 with "LLM not configured".
POST /api/v1/extract returns HTTP 503 — the frontend silently falls back to its built-in rule-based parameter parser, so full-text search continues to work.
All document retrieval and indexing endpoints are completely unaffected.

Taxonomy

config/taxonomy.yaml is the single source of truth for valid filter values. Unknown values are rejected at index time with HTTP 400.

version: "2026-05-19-r1"

knowledge_types:
  - alarm        # 机台报警
  - setup        # 机台 setup / 调试规范
  - experience   # 设备经验 / 故障案例

projects:
  - Kinneret
  - MEM
  - MHK
  - PDX
  - Boston
  - Sonora
  - Yucatan
  - 所有项目    # cross-project documents

equipment:
  - Aligner
  - Conveyor
  - FTU
  - Heater
  - Loader
  - Pump
  - SensorModule
  - Stage

To add a new project or equipment: edit taxonomy.yaml, bump version, reload, then re-seed:

# 1. Reload taxonomy (no restart required)
curl -X POST http://localhost:8000/api/v1/admin/reload-taxonomy

# 2. If you also added new CSV rows: delete the affected index and restart
#    (see "How to update the knowledge base" above)

GET /api/v1/facets returns the live taxonomy — upstream systems call this on startup to know the valid filter values.

Document Types

Every document has common base fields:

Field	Type	Description
`knowledge_type`	enum	`alarm` \| `setup` \| `experience`
`project`	string	Project code from taxonomy
`equipment`	string	Equipment name from taxonomy
`error_codes`	string[]	Optional alarm/error codes (`[A-Z0-9][A-Z0-9_-]{0,63}`)
`title`	string	Max 200 chars; boosted 3× in BM25
`source_file`	string?	Source document filename
`source_pages`	string[]	Page references in source doc

AlarmDoc (`knowledge_type: alarm`)

Field	Required	Description
`content`	yes	Alarm description and context
`resolution`	yes	Step-by-step resolution procedure
`notes`	no	Warnings and additional notes

SetupDoc (`knowledge_type: setup`)

Field	Required	Description
`procedure`	yes	Setup steps
`prerequisites`	no	Required conditions before setup
`notes`	no	Warnings and additional notes

ExperienceDoc (`knowledge_type: experience`)

Field	Required	Description
`body_text`	yes	Free-form experience content
`procedure`	no	Step-by-step procedure (if applicable)
`notes`	no	Warnings and additional notes

API Reference

Method	Path	Description
`GET`	`/`	Knowledge Base Search frontend (HTML)
`GET`	`/healthz`	Liveness check
`GET`	`/api/v1/facets`	Return live taxonomy (projects, equipment, types)
`GET`	`/api/v1/documents/stats`	Aggregate document counts by type/project/equipment
`POST`	`/api/v1/admin/reload-taxonomy`	Hot-reload `taxonomy.yaml` without restart
`POST`	`/api/v1/search`	Search documents (hybrid BM25 + kNN)
`POST`	`/api/v1/documents/{knowledge_type}`	Index a single document
`POST`	`/api/v1/documents/{knowledge_type}/_bulk`	Index multiple documents
`DELETE`	`/api/v1/documents/{knowledge_type}/{doc_id}`	Delete a document
`POST`	`/api/v1/ingest/upload`	Upload files for import (multipart)
`POST`	`/api/v1/ingest/scan`	Scan a server-side folder for import
`GET`	`/api/v1/ingest/sessions`	List recent import sessions
`GET`	`/api/v1/ingest/sessions/{id}`	Get session status + extracted docs
`POST`	`/api/v1/ingest/sessions/{id}/commit`	Commit accepted docs to ES

Full schema available at http://localhost:8000/docs (Swagger UI) or http://localhost:8000/redoc.

Index a document

curl -X POST http://localhost:8000/api/v1/documents/alarm \
  -H "Content-Type: application/json" \
  -d '{
    "project": "MEM",
    "equipment": "Sphere",
    "error_codes": ["125002", "124000"],
    "title": "穿梭真空感应失败",
    "content": "穿梭真空报警分为两种...",
    "resolution": "1. 确认对应报警穿梭穴位...",
    "notes": "注意: 操作前先确认安全状态"
  }'

Search

curl -X POST http://localhost:8000/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{
    "knowledge_type": "alarm",
    "project": "MEM",
    "equipment": "Sphere",
    "error_codes": ["125002"],
    "keywords": ["真空", "穿梭"],
    "query_text": "穿梭真空感应失败怎么处理",
    "mode": "auto"
  }'

Search Behaviour

Search modes

Mode	Keyword logic	Use when
`strict`	AND — document must contain all keywords	Default; precise queries
`loose`	OR — document needs at least one keyword	Fallback; broader recall
`vector_only`	No keyword filter; pure vector similarity	Semantic queries with no exact terms
`auto`	Tries strict → loose → vector in sequence	Default mode

Response status contract

The status field tells the caller how to render the results. It is a required contract — callers must honor it:

Status	Meaning	Required UI behaviour
`strict_hit`	All filters and AND-keywords matched, within threshold	Show results as authoritative
`too_many`	Strict matched > `strict_max_hits`	Do not show docs; prompt user to narrow filters
`loose_hit`	Fell back to OR-keywords	Show with "仅供参考" banner (for reference only)
`vector_only`	Only vector similarity matched	Show with low-confidence banner
`no_hit`	Nothing matched	Inform user; no results

Round-trip parameter echo

Every response includes effective_params — the normalized filter values actually applied. The upstream chat layer should display this to the user (e.g., "您询问 MEM 项目、Sphere 机台…") so they can immediately catch any misextraction.

When status == too_many, the response also includes facets — hit counts by project/equipment — so the caller can suggest which dimension to narrow.

File Import Pipeline

In addition to manually editing CSV files, documents can be imported from PDF, XLSX/XLS, CSV, PPTX, and DOCX files through the web UI or API. The pipeline extracts text, uses the LLM to segment it into structured documents, and lets the user preview/edit before committing to Elasticsearch.

Installing import dependencies

The import pipeline requires extra libraries (pymupdf, openpyxl, python-pptx, python-docx, Pillow):

pip install -e ".[ingest]"
# or: uv sync --extra ingest

For OCR fallback on scanned PDFs, install PaddleOCR separately:

pip install paddlepaddle paddleocr

How it works

Upload/Scan → Hash & Dedup → Extract Text (+OCR) → LLM Segment → Conflict check + Cross-ref → Preview/Resolve/Edit → Commit to ES

Upload files via the web UI (drag & drop) or scan a server-side folder via API
Duplicate detection: SHA-256 file hashes are checked against kb_import_files — previously imported (identical) files are skipped
Text extraction: each file type has a dedicated extractor preserving page boundaries; scanned PDFs fall back to PaddleOCR
LLM segmentation: the configured LLM identifies document boundaries (e.g., which pages belong to which alarm code) and maps extracted text to structured fields (content, resolution, procedure, etc.). Near-duplicate segments are grouped (not dropped) so you can compare variants
Anti-fabrication check: extracted fields are verified against raw source text — discrepancies are flagged in the preview
Conflict detection & cross-referencing: each staged doc whose identity (doc_id) already exists in the KB is flagged as a collision (committing would overwrite it) and blocked until resolved; related existing docs (same error code / equipment / similar) are attached for reference
Preview, resolve & edit: the web UI shows all extracted documents with confidence scores, editable fields, accept/reject checkboxes, a Compare & resolve panel for collisions (keep / overwrite / field-level merge), and a pre-commit summary banner
Commit: accepted documents are validated against taxonomy, embedded (if available), and indexed into Elasticsearch — a keep collision is skipped, overwrite/merge replace the existing doc, and an unresolved collision is reported rather than silently overwritten
Persistence: uploaded files are saved to disk (data/uploads/); committed document payloads are stored in kb_import_files for auto-restore

Auto-restore after restart

CSV seeding still clears all indices on every restart (existing behaviour). After seeding, restore_imports() automatically re-indexes all previously committed imported documents from the kb_import_files tracking index. No manual re-import is needed.

Import API endpoints

Method	Path	Description
`POST`	`/api/v1/ingest/upload`	Upload files (multipart form data)
`POST`	`/api/v1/ingest/scan`	Scan a server-side folder
`GET`	`/api/v1/ingest/sessions`	List recent import sessions
`GET`	`/api/v1/ingest/sessions/{id}`	Get session status and extracted documents
`GET`	`/api/v1/ingest/sessions/{id}/summary`	Pre-commit consequence counts (new/overwrite/keep/unresolved)
`PUT`	`/api/v1/ingest/sessions/{id}/documents/{idx}`	Edit a staged document
`PATCH`	`/api/v1/ingest/sessions/{id}/documents/{idx}`	Accept or reject a document
`PATCH`	`/api/v1/ingest/sessions/{id}/documents/{idx}/resolve`	Resolve a collision (keep / overwrite / merge)
`POST`	`/api/v1/ingest/sessions/{id}/commit`	Commit accepted documents to ES

Upload example

curl -X POST http://localhost:8000/api/v1/ingest/upload \
  -F "files=@alarm_codes.pdf" \
  -F "knowledge_type_hint=alarm" \
  -F "project_hint=MHK" \
  -F "equipment_hint=Loader"

Folder scan example

curl -X POST http://localhost:8000/api/v1/ingest/scan \
  -H "Content-Type: application/json" \
  -d '{
    "folder_path": "/data/import_files",
    "recursive": true,
    "knowledge_type_hint": "alarm",
    "project_hint": "MHK"
  }'

Configuration

Env var	Default	Description
`KB_INGEST__UPLOAD_DIR`	`data/uploads`	Directory for persisting uploaded files
`KB_INGEST__MAX_FILE_SIZE_MB`	`50`	Maximum file size per upload
`KB_INGEST__OCR_ENABLED`	`true`	Enable PaddleOCR fallback for scanned PDFs
`KB_INGEST__SEGMENTATION_MAX_TOKENS`	`4000`	Max LLM response tokens for segmentation
`KB_INGEST__SESSION_TTL_MINUTES`	`120`	Auto-expire staging sessions

Development

Setup

uv sync --extra dev --extra ingest    # app + dev tools (pytest, ruff, mypy) + import libs

Start the API from source with auto-reload (see Option B for the full local setup, including the Elasticsearch container):

uv run python -m kb --reload

Tests, lint, type-check

# Unit tests — fast, no infrastructure required
uv run pytest tests/unit

# Integration tests — requires Docker (Elasticsearch via testcontainers)
uv run pytest tests/integration -m integration

# All tests
uv run pytest

# Lint
uv run ruff check src tests

# Type check (strict)
uv run mypy src

Running multiple instances

Run side-by-side servers on different ports — useful for comparing models or settings:

# Terminal 1 — model A on port 8000
KB_LLM__MODEL=qwen-plus  KB_SERVER__PORT=8000 uv run python -m kb

# Terminal 2 — model B on port 8001
KB_LLM__MODEL=qwen-turbo KB_SERVER__PORT=8001 uv run python -m kb

Editing the seed data

The three CSVs in config/ are re-seeded into ES on every startup, so changes take effect on the next restart — uv run python -m kb locally, or docker compose restart app under Docker. See Data Source — CSV Files for the column mappings.

Building the app image manually

docker compose builds it for you, but you can build the image directly:

docker build -t kb-app .                          # slim (~440 MB, no OCR)
docker build -t kb-app --build-arg INSTALL_OCR=true .   # with PaddleOCR (~2 GB)

The build uses a multi-stage Dockerfile: uv sync --frozen resolves dependencies from uv.lock into a venv, which is copied into a slim Python 3.12 runtime along with config/ and the frontend HTML. .dockerignore keeps the build context lean.

Project Structure

knowledgebase/
├── config/
│   ├── settings.yaml            # Runtime config — ES URL, embedding, search tuning
│   ├── taxonomy.yaml            # Valid projects / equipment / types (edit to extend)
│   ├── 机台报警_header.csv       # Machine alarm documents (100 rows)
│   ├── 机台setup_header.csv      # Equipment setup / calibration documents (100 rows)
│   └── 设备经验_header.csv       # Field experience / failure case documents (100 rows)
├── src/kb/
│   ├── __main__.py              # Entry point: python -m kb [--port PORT] [--host HOST] [--reload]
│   ├── main.py                  # FastAPI app + lifespan (creates indices, seeds, restores imports)
│   ├── config.py                # Pydantic settings (settings.yaml + KB_* env vars)
│   ├── api/
│   │   ├── documents.py         # Index / delete / stats endpoints
│   │   ├── search.py            # POST /api/v1/search
│   │   ├── chat.py              # POST /api/v1/chat + /extract (LLM conversational search)
│   │   ├── facets.py            # GET /api/v1/facets + taxonomy reload
│   │   ├── ingest.py            # File import: upload, scan, preview, edit, commit
│   │   └── deps.py              # FastAPI dependency injection
│   ├── models/
│   │   ├── document.py          # AlarmDoc, SetupDoc, ExperienceDoc (Pydantic v2)
│   │   ├── search.py            # SearchRequest, SearchResponse, SearchStatus
│   │   ├── taxonomy.py          # Taxonomy, KnowledgeType
│   │   └── ingest.py            # ImportSession, StagedDocument, API shapes
│   ├── services/
│   │   ├── csv_loader.py        # CSV files → KnowledgeDoc list
│   │   ├── seed.py              # Startup seeder (CSV → ES) + restore_imports()
│   │   ├── indexing.py          # Document validation + ES bulk indexing
│   │   ├── search.py            # Hybrid search pipeline (strict → loose → vector)
│   │   ├── embedding.py         # DashScope embeddings client (OpenAI-compat)
│   │   ├── taxonomy.py          # TaxonomyStore with hot-reload
│   │   ├── extraction.py        # Per-filetype text extraction (PDF/XLSX/PPTX/DOCX/CSV)
│   │   ├── ocr.py               # PaddleOCR wrapper for scanned PDF fallback
│   │   ├── segmentation.py      # LLM-based document segmentation + anti-fabrication
│   │   ├── import_pipeline.py   # Import orchestrator: hash → extract → segment → commit
│   │   └── file_tracker.py      # File hash tracking in ES (kb_import_files index)
│   └── es/
│       ├── client.py            # Async Elasticsearch client factory
│       ├── mappings.py          # Index mappings (dense_vector + keyword + text fields)
│       ├── import_mappings.py   # Index mapping for kb_import_files (import tracker)
│       ├── body_builder.py      # Builds the ES `body` field from document sections
│       └── migrations.py        # Index create / delete CLI
├── tests/
│   ├── unit/                    # Pure Python, no infrastructure required
│   └── integration/             # Requires Docker (testcontainers + Elasticsearch)
├── elasticsearch/
│   └── Dockerfile               # Custom ES 8.15.3 image — installs the IK analyzer plugin
├── Knowledge Base Search.html   # Single-file React frontend (served at GET /)
├── .env.example                 # Template for .env — copy and fill in KB_LLM__API_KEY
├── .gitattributes               # Enforce LF line endings for all text files
├── Dockerfile                   # App image (multi-stage uv build; INSTALL_OCR build arg)
├── .dockerignore                # Keeps the app build context lean
├── docker-compose.yml           # Full stack: Elasticsearch (+IK) + the app service
├── CLAUDE.md                    # AI agent quick-reference (architecture, commands, constraints)
└── pyproject.toml               # Python dependencies and tool config (uv / pip)

Name		Name	Last commit message	Last commit date
Latest commit History 35 Commits
config		config
docs		docs
elasticsearch		elasticsearch
scripts		scripts
src/kb		src/kb
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
Knowledge Base Search.html		Knowledge Base Search.html
README.md		README.md
docker-compose.yml		docker-compose.yml
mkdocs.yml		mkdocs.yml
pyproject.toml		pyproject.toml
thoughts.md		thoughts.md
uv.lock		uv.lock

Folders and files

Latest commit

History

Repository files navigation

Knowledge Base — Manufacturing Search Engine

Table of Contents

Documentation Site

Auto-publish on push (no CI minutes)

Architecture Overview

Prerequisites

Quick Start

Option A — Docker Compose (recommended)

Option B — Local (host Python + ES container)

What happens on startup

Troubleshooting startup

Data Source — CSV Files

File overview

Column mapping

How to update the knowledge base

Configuration

Common env-var overrides

TLS fingerprint (production)

AI Chat API

Default provider — DeepSeek

Setting your API key

Switching to a different AI provider

All LLM environment variables

Behaviour when no API key is configured

Taxonomy

Document Types

AlarmDoc (knowledge_type: alarm)

SetupDoc (knowledge_type: setup)

ExperienceDoc (knowledge_type: experience)

API Reference

Index a document

Search

Search Behaviour

Search modes

Response status contract

Round-trip parameter echo

File Import Pipeline

Installing import dependencies

How it works

Auto-restore after restart

Import API endpoints

Upload example

Folder scan example

Configuration

Development

Setup

Tests, lint, type-check

Running multiple instances

Editing the seed data

Building the app image manually

Project Structure

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

AlarmDoc (`knowledge_type: alarm`)

SetupDoc (`knowledge_type: setup`)

ExperienceDoc (`knowledge_type: experience`)