context-forge

Self-hosted context infrastructure for AI coding agents. Exposes a single MCP endpoint that gives Claude Code, Codex, Cursor, and other MCP clients persistent memory, semantic code search across repositories, live database schema context, remote files over SSH, reviewed write paths (SQL, files, pull requests), and async job execution. Multi-tenant by design (organizations and projects), managed from a web UI.

Features

• Persistent memory — long-term memory with Mem0 + pgvector, scoped per organization and project.
• Knowledge base — upload documents (PDF, Word, Excel, PowerPoint, images with OCR, text, and more) via drag-and-drop, or fetch them from a URL (with SSRF guards); they're extracted, chunked, embedded, and made semantically searchable.
• Organization catalog — the organization owns its resources and each project selects what it uses. Machines hold SSH credentials once per organization; SSH folders, databases and repositories are registered in the catalog (by org admins from the UI, or through the *_add MCP tools) and then selected per project. A resource marked restricted can only be selected by an org admin. Deleting a catalog entry removes it from every project that selected it.
• Hybrid repository search — index and query local, GitHub, and GitLab repos (gitlab.com or self-hosted via GITLAB_URL) using tree-sitter parsing. Retrieval fuses dense vector embeddings with lexical full-text ranking (Reciprocal Rank Fusion) so exact identifiers, error strings, and rare tokens surface alongside semantic matches. Repositories live in the organization catalog and each project selects the ones it searches. Set SEARCH_HYBRID=false to fall back to vector-only.
• Code intelligence — a symbol graph (definitions, references, imports) built at index time powers repo_map (ranked file overview for a query, definitions first), repo_neighbors (callers, callees, and related files) and repo_symbols; repo_get_file reads by line window so agents pull only the lines they need.
• Data sources and database scopes — register database connections (PostgreSQL, MySQL, MariaDB, SQLite) in the catalog, directly or through an SSH machine tunnel. A project links a connection on a scope — a database and, on PostgreSQL, a schema — under an alias; the same connection can be linked on several scopes. Agents address data sources by alias, and queries and writes are confined to the scope's schema. Agents get live schema context (tables, columns, keys, indexes, row estimates) enriched by a human-curated data dictionary (per-table and per-column descriptions edited in the UI), and can run validated read-only SQL (single SELECT/SHOW/EXPLAIN statement, enforced LIMIT, timeouts, full audit log). With the db-write permission, db_execute runs a single validated DML statement (real top-level WHERE required) inside a transaction with a DB-side timeout; without it, a caller holding context-write gets a proposal instead — the statement is stored with an EXPLAIN preview for an admin to approve from the Approvals page, pinned to the scope it was previewed on: approval fails if that scope was renamed, removed or re-pointed in the meantime. Credentials are encrypted at rest when ENCRYPTION_KEY is set.
• SSH file sources — register an SSH folder (a catalog machine plus a root directory) and select it in the projects that need it; agents list, read (head/tail/offset windows), and grep files there, and with ssh-write can atomically write files confined to that root. Without ssh-write, a caller holding context-write proposes the write instead: the content is stored with a size/hash preview for an admin to approve, pinned to the folder it was previewed on: approval fails if that folder was renamed, removed, or moved to another machine or root in the meantime.
• Code changes — repo_commit_files and repo_open_pr clone a scratch workspace, commit agent-proposed changes on a branch, push, and open a GitHub PR / GitLab MR for human review.
• API contracts — ingest OpenAPI/Swagger specs (URL or pasted JSON/YAML) and GraphQL schemas (live introspection or pasted result). Every operation becomes individually listable and searchable, with $ref-resolved request/response schemas, so agents know exactly which endpoints exist and with what payloads.
• CI/CD context — live view of recent GitHub Actions / GitLab CI runs for configured repos, and a "why is the pipeline red" tool that returns the failed jobs/steps with the ANSI-stripped tail of their error logs. Uses the tokens already configured for indexing; nothing extra to set up.
• Async jobs — offload slow downstream calls without hitting client timeouts. Jobs are executed by the scheduler, so they survive a restart; transient failures are retried with exponential backoff and exhausted jobs land in a dead state.
• Agent chat — a built-in chat page where a tool-using agent searches your repos, memory, knowledge base, and connected databases, showing every retrieval inline so you can verify context is surfaced correctly. Streams responses (and model reasoning when available), switches models on the fly across the providers you have keys for (OpenAI, Anthropic, DeepSeek), keeps a per-user chat history, and can publish a conversation snapshot as a public share link.
• Multi-tenancy — organizations as isolation boundaries with owner / admin / member / viewer roles. Each organization holds one or more projects that scope every source, search, and memory namespace; users are created by org admins with per-project visibility. Provider keys, embedding model, git tokens, and the Telegram capture bot can all be overridden per organization. Every MCP tool call is audited per organization, and API keys can carry a per-minute rate limit.
• Authentication — local admin login, or any OIDC provider (Keycloak, Authentik, Entra ID, ...) with just-in-time user provisioning. The MCP endpoint accepts API keys with granular permissions or OIDC bearer tokens, and can act as an OAuth authorization server so MCP clients log in from the browser without an API key.
• Runtime-first config — manage the catalog, providers, tokens, and indexing from the UI; .env and YAML are only for bootstrap.
• Pluggable providers — OpenAI, Jina, OpenAI-compatible, or fully local embeddings.

Architecture

Service	Stack	Ports
context-forge	Python 3.11, FastAPI (REST), FastMCP (MCP)	8000/api, 4000/mcp
postgres	PostgreSQL 16 + pgvector	5440 on the host → 5432 in the container
ui	React 18, TypeScript, Vite, TailwindCSS	3000

Indexing uses tree-sitter (Python, JS/TS, Go, Java), with scheduled re-indexing via APScheduler. Re-indexing is incremental: for git-backed repos, only files changed since the last indexed commit are re-parsed and re-embedded, and the symbol graph is rebuilt for them. Pushes can trigger it immediately via the /api/webhooks/index endpoint (set WEBHOOK_SECRET; supports GitHub, GitLab, and generic callers).

Operations are observable from two endpoints. GET /metrics serves Prometheus text exposition on the API port with the queue depths (jobs by status, pending index requests and the age of the oldest one, pending knowledge-base documents and web pages), the scheduler heartbeat, the applied schema version, MCP tool-call counters, and the depth of the MCP audit write queue (contextforge_mcp_audit_queue_depth); the setup wizard generates a random METRICS_TOKEN, so a scrape must send it (curl -H "Authorization: Bearer $METRICS_TOKEN" http://localhost:8000/metrics); emptying it in .env leaves the endpoint open, which is only reasonable on a private scraping network. /metrics is served by the API port (8000) and is not proxied by the UI's nginx, which forwards /api/ only. GET /api/health/details returns the same picture as JSON for organization admins and owners, plus the database latency, and reports degraded when the database check fails, the queue or schema-version check fails, the scheduler has not ticked for more than 60 seconds (or never ticked), or the oldest index request is more than an hour old. GET /api/health stays public and minimal. No metric is labelled by organization.

Vector search is served by one HNSW index per organization and embedding dimension, built on the typed cast embedding::vector(N) and partial on org_id, so organizations on different embedding models stay independently indexed. The indexes are created and pruned in the background at startup, when embedding settings are saved without changing the dimension, and every six hours as a self-healing pass; a re-embed instead drops them before the first batch and rebuilds them inline at the end, because vectors of the new width do not fit the old index. Builds run on a dedicated connection with no command timeout and HNSW_MAINTENANCE_WORK_MEM (default 256MB) of work memory, and an index that an interrupted build left INVALID is dropped and rebuilt by the next pass.

Quick start

bash setup.sh            # or: .\setup.ps1 on Windows
docker compose up -d

The setup wizard creates .env from .env.example and generates random POSTGRES_PASSWORD, SETUP_BOOTSTRAP_TOKEN and METRICS_TOKEN values; you only need to add your OPENAI_API_KEY (or another embeddings provider).

Then open the UI at http://localhost:3000. On first run the onboarding wizard asks for the bootstrap token — the SETUP_BOOTSTRAP_TOKEN value in your .env (it proves that whoever creates the admin account has access to the server environment) — and for the credentials of the web UI admin account.

Minimum environment variables:

POSTGRES_PASSWORD=...       # generated by the wizard
SETUP_BOOTSTRAP_TOKEN=...   # generated by the wizard
OPENAI_API_KEY=...          # or another configured provider

See .env.example for the full list (MCP auth, OIDC, webhooks, credential encryption); every variable is passed through to the container by docker-compose.yml.

Upgrading an existing installation

The schema migrates itself at startup. Coming from a version without projects and OIDC:

• every organization gets a Default project, and all existing sources, chunks, and memories are attached to it;
• the former local OAuth server for MCP is gone: tokens it issued stop working, and MCP clients should use an API key or the OIDC bridge instead;
• API keys keep working; their legacy scope is mapped to the new permissions (read → context-read, write → context-read + context-write, admin → *);
• global LLM/provider overrides move into the per-organization settings;
• embedding columns lose their fixed dimension so each organization can pick its own model. The old ivfflat indexes are dropped in the process and replaced by per-organization HNSW indexes, which the server builds in the background on the first startup after the upgrade; until each build completes vector search runs unindexed on that table (a sequential scan), so it is correct but slow, and the build itself runs on a dedicated connection with no timeout;
• async jobs the previous fire-and-forget executor left running are marked dead on the first start after this upgrade: it may already have delivered them, so they are not re-fired; jobs still pending never ran and are executed normally under the new retry policy;
• on the first start after this upgrade the server records the existing schema as migration version 1 (the frozen baseline is replayed once; it is idempotent), and later changes apply as numbered migrations; python -m src.cli migrate --status shows the current version.

Upgrading to the organization catalog and database scopes

On the first boot after this upgrade three one-way conversions run:

1. SSH credentials are grouped into machines; SSH folders and database connections move to the organization catalog, and projects keep them as selections. Duplicate SSH folder and database names inside an organization are renamed by appending the slug of the project they belonged to (name-<project-slug>, plus a number if that is still taken); the first one keeps its name.
2. Repositories get numeric ids; duplicates with the same URL and branch are merged into one (the duplicates' indexed chunks and clone directories are discarded), and clone directories move to org_<id>/repo_<id> in the repos cache.
3. Project-to-database links become inferred scopes (the connection's database and schema under an alias). Confirm them from the project's data sources page.

The conversions drop the legacy columns and tables (if some database links cannot be converted, the legacy link table is kept and the server log lists them) and cannot be undone: back up the database and the repos cache directory before upgrading. A legacy SSH secret that cannot be decrypted with the current ENCRYPTION_KEY stops the boot; fix the key and start again.

Other changes to be aware of:

• The REST endpoints that created, updated or deleted resources per project are gone: writes on /api/repos, /api/ssh-sources and /api/datasources, and /api/github/repos/add / /api/gitlab/repos/add. Use /api/catalog/* (machines, folders, databases, repos) instead; the read, index, file and query endpoints remain, and the data-source endpoints take a scope id.
• Agents can no longer query information_schema or pg_* through db_query; use db_schema and db_describe to learn the structure.
• ssh_source_add, db_add and repo_add register in the catalog and select for the active project; a resource that exists in the catalog but not in the project is found with catalog_list and added with resource_select.
• Write proposals (db_execute, ssh_write_file) still pending before the upgrade fail on approval and must be proposed again.

Development

Schema migrations

The schema is versioned. Every change is a new module under services/server/src/migrations/versions/, named NNNN_<name>.py:

"""Add the retry bookkeeping columns to jobs."""
from __future__ import annotations

VERSION = 2
NAME = "jobs_retry"
TRANSACTIONAL = True


async def upgrade(conn) -> None:
    await conn.execute(
        "ALTER TABLE jobs ADD COLUMN IF NOT EXISTS attempts INT NOT NULL DEFAULT 0"
    )

• VERSION is the file number and NAME the file suffix; both are recorded in the schema_migrations table.
• Modules run once, in VERSION order, at server startup, on a single connection holding a Postgres advisory lock, so concurrent replicas cannot race. A module that raises aborts the boot; nothing after it runs.
• TRANSACTIONAL = True wraps the module in a transaction. Set it to False only when the statement cannot run inside one (CREATE INDEX CONCURRENTLY); such a module must be idempotent, because a failure leaves it half-applied and unrecorded.
• Never edit services/server/src/db.py:DDL. It is frozen as versions/0001_baseline.py, and every existing installation has already applied it.

From services/server:

.venv/Scripts/python.exe -m src.cli migrate --status   # current version and pending modules
.venv/Scripts/python.exe -m src.cli migrate            # apply pending modules
.venv/Scripts/python.exe -m pytest -q -p no:cacheprovider

Both migrate forms connect with DATABASE_URL, the same setting the server uses. The CLI applies the versioned schema modules only; the data conversions described in the upgrade notes run at server boot.

Tests against PostgreSQL

Tests that need a real PostgreSQL are skipped unless TEST_DATABASE_URL points at a PostgreSQL 16 server with the pgvector extension (for example the postgres service of docker-compose.yml). Each test creates and drops its own throwaway database on that server, so the URL only needs an account allowed to create databases:

TEST_DATABASE_URL=postgresql://context_forge:<password>@localhost:5440/postgres \
  .venv/Scripts/python.exe -m pytest -q -p no:cacheprovider

Projects and the MCP endpoint

An organization can hold several projects. Connect to the organization-level endpoint /mcp/<org-slug> and pick a project in-session with list_projects / use_project (memory, search, and every scoped tool follow that choice), or point a client straight at /mcp/<org-slug>/<project-slug>. API keys can be restricted to a set of projects, and org admins choose which projects each user can see.

MCP tools

• Memory: memory_add, memory_search, memory_list, memory_delete
• Knowledge base: kb_search, kb_list, kb_get_document, kb_add_url
• Catalog: catalog_list (context-read) lists the organization's SSH folders, databases and repositories and whether the active project selected each; resource_select and resource_deselect (context-write plus membership of the project; restricted resources need an org admin) add or remove a resource from the project. For databases, resource_select takes database, schema and alias to link a scope.
• Repositories: repo_list, repo_search, repo_symbols, repo_get_file, repo_index, repo_relationships, repo_add (sources-write; registers the repository in the catalog if needed and selects it, reusing one with the same URL and branch)
• Code intelligence: repo_references, code_explain, repo_annotate, repo_annotations, repo_map, repo_neighbors
• Code changes: repo_commit_files, repo_open_pr — create a branch, push agent-proposed file changes, and open a GitHub PR / GitLab MR for human review (requires the repo-write permission; git tokens need write scope)
• Data sources: db_list, db_schema, db_describe, db_query, db_execute (db-write, otherwise a proposal), db_add (sources-write; registers a catalog connection, optionally through an SSH machine, or reuses one to the same server, and links a database/schema scope; a new connection through a catalog machine, or on SQLite, requires an organization admin). The db_* tools take the scope alias as connection (a scope id also works); db_list shows the aliases.
• SSH files: ssh_sources, ssh_list_files, ssh_read_file, ssh_grep, ssh_write_file (ssh-write, otherwise a proposal), ssh_source_add (sources-write; takes a catalog machine, or host and username for a new one, plus the root path; creating a new folder on a machine already in the catalog requires an organization admin; API keys and, with MCP auth disabled, anonymous callers never count as one, so they register such folders and connections from the Catalog page)
• Write approvals: write_request_status — check whether a proposed write was approved, rejected, or executed
• Web pages: web_search, web_list, web_get_page, web_add, web_refetch, web_list_sites, web_crawl, web_delete, web_delete_site
• API contracts: api_list, api_endpoints, api_get_endpoint, api_add
• CI/CD: ci_runs, ci_failure
• Jobs: job_submit, job_status, job_result
• Projects: list_projects, use_project, current_project; with projects-write: create_project, rename_project, delete_project, add_project_member

New sources registered through MCP (db_add, ssh_source_add) are created without their secret and stay in pending_secret until an admin completes them from the Catalog page; selecting a machine, folder or connection that already exists in the catalog creates nothing pending.

Agent setup

Connection guides per client:

• Claude Code
• Codex
• Cursor

Claude Code with OIDC (browser login)

With OIDC_ENABLED=true and MCP_AUTH_MODE=enabled, the MCP endpoint acts as an OAuth bridge to your identity provider: add the server without an API key and authenticate through the provider's login page in the browser. The endpoint is organization-level (/mcp/<org>); pick the project afterwards with the use_project tool.

claude mcp add --transport http --scope user context-forge https://<host>/mcp/<org>

No --header: on first use Claude Code detects that authentication is required, opens the browser on the provider's login page, and receives the token.

MCP servers are loaded when a session starts: after claude mcp add (and after the login) the server and its tools show up in a new Claude Code session. In a session that is already open, use /mcp → reconnect to trigger the login right away.

Alternatively, with an API key (no browser, suited to headless agents):

claude mcp add --transport http --scope user context-forge https://<host>/mcp/<org> \
  --header "X-API-Key: forge_..."

Clients that can only set an Authorization header may send the key as Bearer forge_....

Security

The REST API/UI is authenticated after setup (local admin login, or OIDC when OIDC_ENABLED=true). The MCP endpoint has three modes, selected with MCP_AUTH_MODE:

• disabled — no authentication (local development only);
• transition — API keys and OIDC tokens are honored when present; anonymous callers only get read-only tools;
• enabled — every request must carry an API key or an OIDC bearer token.

Expose the MCP port only behind TLS with MCP_AUTH_MODE=enabled. Data-source and SSH credentials are encrypted at rest when ENCRYPTION_KEY is set; git and OIDC tokens are never echoed in error messages.

Every MCP tool call is recorded: who called it (OIDC user, API key, or anonymous), the tool, the project, the outcome (ok, denied, error, rate_limited), the duration, and a redacted summary of the arguments — credentials, tokens, secrets and payloads are redacted; URLs are stored without userinfo or query strings, while audited SQL is kept truncated. A tool that answers with an error instead of raising is recorded as error too, so the Errors tile counts every call that reported one. Org admins and owners read the trail under Organization → Tool activity, with a 24h/7d stats strip and filters by tool, by outcome, and by principal (substring match). History is pruned daily according to MCP_AUDIT_RETENTION_DAYS (default 90).

SQL validation and scopes

Every statement sent to a data source goes through a validator before it reaches the database. Besides the single-statement rules (read-only, or a single DML statement with a real WHERE), it rejects the system catalogs (information_schema, pg_catalog, relations named pg_*), SELECT ... INTO, and file, session, large-object and dblink functions, and it confines qualified names to the scope's schema. Agents learn the structure of a data source with db_schema and db_describe, not from the catalogs. Pooled connections are reset when they return to the pool, so session state set by one call never reaches the next.

Human approvals for writes

db_execute and ssh_write_file execute immediately only for a caller holding db-write / ssh-write. A caller holding context-write but not the write permission does not execute: the tool stores a write request — the statement or the file content, plus a preview (estimated rows and plan excerpt for SQL, sizes and sha256 hashes for files) — and returns pending_approval with a request id to poll via write_request_status. A caller with neither permission is denied as before.

Organization admins and owners review the queue under Approvals in the dashboard. Approving requires the same write permission the request needs (owners always qualify), and the change then runs under the approver's identity. Pending requests expire after 7 days; a daily job marks them expired.

MCP permissions

The identity provider only authenticates. Authorization of MCP tools is managed in the application database: each organization role (viewer, member, admin, owner) has a set of permissions, editable from the dashboard under Organization → MCP permissions.

Permission	Grants
context-read	search and read memory, repositories, knowledge base, web pages, API contracts, CI runs, database schemas
context-write	add or delete memories, annotate code, add and refetch web pages, select and deselect catalog resources (resource_select, resource_deselect; requires membership of the project, and restricted resources need an org admin); also propose db_execute / ssh_write_file writes for an admin to approve
db-query	read-only SQL on connected databases
db-write	db_execute (validated single-statement DML) and approving SQL write requests
repo-write	repo_commit_files, repo_open_pr
jobs	submit and read async jobs
ssh-read / ssh-write	read / write files on SSH sources; ssh-write also approves file write requests
projects-write	create, rename, delete projects and manage their members
sources-write	register repositories, API contracts, knowledge-base URLs, databases, and SSH folders in the catalog
*	everything, including re-indexing and web crawling/deletion

catalog_list needs context-read; no permission names were added with the catalog. Defaults: viewer → context-read; member → context-read, context-write, db-query, ssh-read; admin → the member set plus jobs, projects-write, sources-write; owner → *. Write capabilities (db-write, ssh-write, repo-write) are owner-only until granted explicitly. Members holding context-write can still propose SQL and file writes; an admin with the matching write permission approves them under Approvals.

API keys get granular permissions chosen at creation (Settings → MCP keys), never more than the creator's own role allows, plus an optional rate limit in calls per minute (empty = unlimited; enforced per server process), editable afterwards from the key list. OIDC groups under OIDC_GROUP_PREFIX are only used to suggest the initial role when a user first logs in.

License

MIT