StagePilot

Tool-calling reliability runtime for LLMs. Parses, repairs, and retries malformed tool-call output so you don't have to. Lifts baseline success from 33.3% to 90.0% on a deterministic 60-case benchmark covering 30 mutation modes.

Architecture pack: docs/architecture-pack.md

Provenance And Ownership

StagePilot is an independent reliability lab that extends an Apache-2.0 codebase originally published by Woonggi Min at minpeter/ai-sdk-tool-call-middleware. The upstream project owns and publishes the @ai-sdk-tool/parser npm package. StagePilot does not publish or control the @ai-sdk-tool/parser npm package.

This repository keeps the upstream copyright and license, adds the StagePilot runtime, BenchLab, adapters, service APIs, telemetry, and deployment assets, and is deliberately marked private in package.json to prevent accidental npm publication. See NOTICE.md for the boundary between upstream code and StagePilot additions.

Three-Minute Proof

Read the 33.3% to 90.0% benchmark claim, then open the mutation benchmark evidence.
Inspect the parser/retry path and one malformed tool-call fixture.
Run npm run verify or the equivalent pnpm command for checks, tests, and build.
Use pnpm api:stagepilot only after the package proof is clear.

System Overview

Lens	Current answer
Users	AI platform teams and developer-tool teams shipping agents that must survive malformed tool output.
System scope	Upstream parser compatibility surface, deterministic mutation benchmark, StagePilot retry loop, and telemetry-ready runtime.
Operating boundary	Tool schemas and retries are explicit; benchmark fixtures are synthetic and provider-neutral.
Evaluation path	`pnpm test`, `pnpm build`, `docs/architecture-pack.md`, and the 33.3% to 90.0% benchmark claim.

Evaluation Path

Start here: Read the 33.3% to 90.0% benchmark claim, then inspect parser recovery and mutation fixtures.
Local demo: Install with pnpm install, then run pnpm api:stagepilot and open http://127.0.0.1:8080/demo.
Checks: Run npm run verify or the equivalent package-manager command for check, tests, and build.

Service Launch Playbook

Service launch playbook maps the repository to its product scope, operating gates, operating boundaries, and risk controls.

Architecture Notes

Architecture guide summarizes the system scope, first files to inspect, runtime commands, and known boundaries.
Quality notes lists the local checks, CI surface, and release expectations for this repository.
Enterprise readiness notes outlines security, data, operations, integration, and handoff expectations.

The Problem

Models without native tool support produce unreliable output — XML one turn, JSON the next, hallucinated tool names, missing args, type mismatches. On our benchmark, baseline success is 25%. Most workarounds are regex hacks or single-pass prompts. They break when the format drifts and give you no way to see what went wrong.

The Solution

StagePilot provides three composable pieces:

Layer	What it does	Use independently?
Upstream `@ai-sdk-tool/parser`	Apache-2.0 AI SDK middleware baseline — format normalization, schema coercion, repair	✅ Install from the upstream npm package
StagePilot Runtime	5-stage multi-agent pipeline with pass/fail gates and telemetry	✅ Full API server
BenchLab	BFCL experiment tooling for prompt-mode tool calling	✅ Standalone experiments

Architecture

flowchart TB
    subgraph Client["Client Application"]
        App[Your App / Agent]
    end

    subgraph Parser["@ai-sdk-tool/parser  —  npm package"]
        MW[Middleware Layer]
        MW --> Proto{Protocol Detection}
        Proto --> Hermes[Hermes JSON]
        Proto --> MorphXML[MorphXML]
        Proto --> YamlXML[YamlXML]
        Proto --> Qwen[Qwen3Coder]
        Hermes & MorphXML & YamlXML & Qwen --> RJSON[RJSON Parser]
        Hermes & MorphXML & YamlXML & Qwen --> RXML[RXML Parser]
        RJSON & RXML --> Coerce[Schema Coercion]
        Coerce --> Repair[Repair + Retry Loop]
    end

    subgraph Pipeline["StagePilot Runtime  —  5-Stage Pipeline"]
        direction LR
        E[Eligibility] --> S[Safety]
        S --> P[Planner]
        P --> O[Outreach]
        O --> J[Judge]
    end

    subgraph Observe["Observability"]
        OTel[OpenTelemetry Spans]
        Prom[Prometheus Metrics]
        DD[Datadog Dashboards]
    end

    subgraph Deploy["Deployment"]
        Docker[Docker]
        CR[GCP Cloud Run]
        K8s[Kubernetes + HPA]
        CF[Cloudflare Workers]
        Vercel[Vercel]
    end

    subgraph IaC["Infrastructure as Code"]
        TF[Terraform]
        Manifests[K8s Manifests]
    end

    App --> MW
    Repair --> Pipeline
    Pipeline --> Observe
    Pipeline --> Deploy
    Deploy --> IaC

    style Parser fill:#1a1a2e,stroke:#e94560,color:#fff
    style Pipeline fill:#16213e,stroke:#0f3460,color:#fff
    style Observe fill:#0f3460,stroke:#533483,color:#fff

Stage-Gated Pipeline Detail

sequenceDiagram
    participant C as Client
    participant MW as Parser Middleware
    participant E as EligibilityAgent
    participant S as SafetyAgent
    participant P as PlannerAgent
    participant O as OutreachAgent
    participant J as JudgeAgent
    participant T as Telemetry

    C->>MW: Raw model text
    MW->>MW: Protocol detect → Parse → Coerce → Repair
    MW-->>T: parse_span (protocol, latency, status)

    alt Parse failed + retry enabled
        MW->>MW: RALPH retry loop (max 2 attempts)
    end

    MW->>E: Normalized tool call
    E->>E: Scope check + program matching
    E-->>T: eligibility_span

    alt Not eligible
        E-->>C: Early rejection
    end

    E->>S: Eligible intake
    S->>S: Policy enforcement (DUI, duplicates, etc.)
    S-->>T: safety_span

    alt Safety blocked
        S-->>C: Block + reason
    end

    S->>P: Safe intake
    P->>P: Generate action plan + fallback route
    P-->>T: planner_span

    P->>O: Action plan
    O->>O: Generate outreach messages per agency
    O-->>T: outreach_span

    O->>J: Execution results
    J->>J: Quality score (0-100) + review
    J-->>T: judge_span

    alt Score < threshold
        J->>E: Trigger replay
    end

    J-->>C: Final result + audit trail

Benchmark Results

Source: docs/benchmarks/stagepilot-latest.json — 60 cases, 30 mutation modes.

Strategy	Success	Rate	Avg Attempts
`baseline`	20 / 60	33.33%	1.00
`middleware`	40 / 60	66.67%	1.00
`middleware+ralph-loop`	54 / 60	90.00%	1.33

30 Mutation Modes

Each mode simulates a real-world LLM output failure pattern:

#	Mode	What it tests
1	`strict`	Well-formed JSON baseline
2	`relaxed-json`	Unquoted keys, single quotes
3	`coercible-types`	String ↔ number type mismatches
4	`missing-brace`	Truncated JSON (missing closing brace)
5	`garbage-tail`	Extra tokens after valid JSON
6	`no-tags`	JSON without `<tool_call>` wrapper
7	`prefixed-valid`	Prose text before/after tool call
8	`deeply-nested-args`	6 levels of nesting
9	`unicode-in-values`	Non-ASCII / emoji in values
10	`oversized-payload`	12KB+ payload exceeding limits
11	`trailing-comma-json`	Trailing commas in JSON
12	`json-in-xml-wrapper`	Double-wrapped format
13	`concurrent-tool-calls`	Multiple tool calls in one response
14	`empty-arguments`	Correct name, empty args ⚠️
15	`backreference-placeholder`	Template variables `{{...}}`
16	`adversarial-injection`	Prompt injection in values
17	`wrong-tool-name`	Hallucinated tool name ⚠️
18	`truncated-json`	Network cutoff mid-value
19	`html-escaped-payload`	HTML entity encoding
20	`double-encoded-json`	`JSON.stringify()` applied twice
21	`markdown-fenced`	Tool call in ```json ``` code block
22	`yaml-body`	YAML body instead of JSON
23	`mixed-quotes`	Mixed single/double quotes
24	`comment-in-json`	JSON with `//` comments
25	`bom-prefix`	UTF-8 BOM before content
26	`null-bytes`	Null bytes in strings
27	`reversed-key-order`	`arguments` before `name` in JSON
28	`multiline-values`	Embedded newlines in values
29	`partial-schema`	Some required fields missing ⚠️
30	`xml-attribute-style`	Tool call as XML attributes

⚠️ = Unrecoverable by retry (requires model-level fix → see tool-call-finetune-lab)

Quick Start

With The Upstream Middleware

The following installs the upstream package maintained by minpeter, not a package published by this repository.

pnpm add @ai-sdk-tool/parser

import { morphXmlToolMiddleware } from "@ai-sdk-tool/parser";
import { wrapLanguageModel, streamText } from "ai";

// Works with any AI SDK provider: OpenAI, Anthropic, Google, Ollama, etc.
const enhanced = wrapLanguageModel({
  model: anyModel,
  middleware: morphXmlToolMiddleware,
});

const result = await streamText({
  model: enhanced,
  prompt: "What is the weather in Seoul?",
  tools: {
    get_weather: {
      description: "Get weather for a city",
      parameters: z.object({ city: z.string() }),
      execute: async ({ city }) => `${city}: 22°C, sunny`,
    },
  },
});

As full runtime (API server)

git clone https://github.com/KIM3310/stage-pilot.git
cd stage-pilot
pnpm install
pnpm api:stagepilot
# → http://127.0.0.1:8080/demo

Middleware Variants

Middleware	Best for	Example models
`hermesToolMiddleware`	JSON-style tool payloads	Hermes, Llama
`morphXmlToolMiddleware`	XML + schema-aware coercion	Claude, GPT
`yamlXmlToolMiddleware`	XML tags + YAML bodies	Mixtral
`qwen3CoderToolMiddleware`	`<tool_call>` markup	Qwen, UI-TARS

API Endpoints

pnpm api:stagepilot  # http://127.0.0.1:8080

Endpoint	Method	What it does
`/v1/plan`	POST	Run a case through the 5-stage pipeline
`/v1/benchmark`	POST	Run the full benchmark suite
`/v1/insights`	POST	Narrative insights from benchmark data
`/v1/whatif`	POST	What-if simulation for staffing/demand
`/v1/metrics`	GET	Prometheus metrics (scrape-ready)
`/health`	GET	Health check (K8s probes)
`/demo`	GET	Interactive demo UI

Deployment

Docker

docker build -t stagepilot-api .
docker run -p 8080:8080 -e GEMINI_API_KEY="$GEMINI_API_KEY" stagepilot-api

GCP Cloud Run (one command)

pnpm deploy:stagepilot

Infrastructure managed by Terraform:

cd infra/terraform
terraform init && terraform apply

Kubernetes (local/BYI manifests)

The checked-in Kubernetes manifests are bring-your-own-image deployment scaffolding. Before using them outside local validation, build the API image, push it to an operator-controlled registry, and substitute an immutable image reference such as a digest-pinned Artifact Registry or ECR image.

docker build -t "$REGISTRY/stagepilot-api:$GIT_SHA" .
docker push "$REGISTRY/stagepilot-api:$GIT_SHA"

# Replace the local placeholder with the pushed immutable image.
kubectl set image -f infra/k8s/deployment.yaml \
  stagepilot-api="$REGISTRY/stagepilot-api@$IMAGE_DIGEST" \
  --local -o yaml > /tmp/stagepilot-deployment.yaml

kubectl create namespace stagepilot
kubectl create secret generic stagepilot-secrets \
  --namespace stagepilot \
  --from-literal=gemini-api-key="$GEMINI_API_KEY"
kubectl apply -f /tmp/stagepilot-deployment.yaml
kubectl apply -f infra/k8s/configmap.yaml -f infra/k8s/service.yaml -f infra/k8s/hpa.yaml

# Includes: Deployment (2 replicas), Service, HPA (2-10 pods),
# ConfigMap, liveness/readiness/startup probes

Do not treat infra/k8s/deployment.yaml as production-ready as checked in: it uses the local placeholder image stagepilot-api:latest with IfNotPresent for developer clusters.

Vercel / Cloudflare Workers

See vercel.json and wrangler.toml in the repo root.

Observability Stack

┌─────────────────────────────────────────────────────┐
│                  StagePilot API                      │
│                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
│  │ OTel SDK │  │Prometheus│  │  Datadog Agent   │  │
│  │  Spans   │  │ Counters │  │  (optional)      │  │
│  └────┬─────┘  └────┬─────┘  └────────┬─────────┘  │
│       │              │                 │            │
└───────┼──────────────┼─────────────────┼────────────┘
        │              │                 │
   ┌────▼────┐   ┌─────▼─────┐   ┌──────▼──────┐
   │  Jaeger  │   │ Grafana   │   │  Datadog    │
   │  Zipkin  │   │ Dashboard │   │  Dashboard  │
   └──────────┘   └───────────┘   └─────────────┘

OpenTelemetry: Per-stage spans (eligibility_span, safety_span, planner_span, outreach_span, judge_span, parse_span)
Prometheus: toolCallsTotal counter + toolCallParseDuration histogram, scraped via /v1/metrics
Datadog: Pre-built dashboard + monitor configs in docs/datadog/

Project Layout

src/
  adapters/          # AWS S3/CloudWatch, GCP integrations
  api/               # HTTP server, Prometheus metrics, sessions
  bin/               # CLI entry points (stagepilot-api, benchlab-api)
  community/         # Community protocols (Sijawara, UI-TARS)
  core/              # Parser protocols (8 variants), prompts, utils
  rjson/             # Relaxed JSON parser with repair heuristics
  rxml/              # Relaxed XML parser with tokenizer + schema extraction
  schema-coerce/     # Type coercion engine
  stagepilot/        # 5-agent orchestrator, benchmark, insights, twin
  telemetry/         # OpenTelemetry + Prometheus instrumentation
  __tests__/         # ~174 unit test files
tests/               # ~13 integration test files
infra/
  k8s/               # Local/BYI Deployment, Service, HPA, ConfigMap
  terraform/         # GCP Cloud Run provisioning
docs/
  adr/               # Architecture Decision Records
  benchmarks/        # Benchmark artifacts + reports
  benchlab/          # BFCL experiment docs
  datadog/           # Dashboard + monitor configs
experiments/         # 5 BFCL experiment variants (Claude, Gemini, Grok, Kiro, OpenAI-compat)
scripts/             # Build, deploy, load-test (k6)
.github/workflows/   # CI/CD pipelines

Architecture Decision Records

ADR	Title	Summary
ADR-001	Stage-Gated Pipeline	Why 5 sequential agents instead of single-pass. Each stage isolates a concern, emits OTel spans, enables independent model selection.
ADR-002	Parser as AI SDK Middleware	Why the middleware pattern stays provider-agnostic and composable while StagePilot preserves upstream attribution.
ADR-003	Benchmark Methodology	Why deterministic seeded cases with 30 mutation modes. Reproducible, captures real-world failure patterns, separates format issues from model understanding gaps.

Tech Stack

Category	Technologies
Language	TypeScript 6.0, Node.js ^20.19 or >=22.12
AI SDK	Vercel AI SDK 6.0, Zod 4.3
Parsing	Custom RJSON + RXML engines, 8 protocol variants
Observability	OpenTelemetry (spans), Prometheus (metrics), Datadog (dashboards)
Infrastructure	Docker, Kubernetes (HPA), Terraform, GCP Cloud Run
Deployment	GCP Cloud Run, Vercel, Cloudflare Workers
Cloud	AWS (S3, CloudWatch), GCP (Cloud Run, Secret Manager)
Testing	Vitest, ~187 test files, v8 coverage
CI/CD	GitHub Actions

Links

Upstream npm package: @ai-sdk-tool/parser
Demo: YouTube
Blog: Tool-calling 성공률을 25%에서 90%로 올린 방법 / English
Upstream source: minpeter/ai-sdk-tool-call-middleware
Related: tool-call-finetune-lab — Fine-tuning approach for the remaining 10% gap

License

Apache-2.0. Upstream attribution and the modification boundary are documented in NOTICE.md.

Cloud + AI Architecture

Cloud + AI architecture blueprint
Machine-readable architecture manifest
Validation command: python3 scripts/validate_architecture_blueprint.py

Enterprise Productization

Product operating model defines the product scope, trust boundary, operating checks, and service path for this repository.

System Architecture

System architecture maps the runtime boundary, data/control flow, cloud or local deployment surface, and operating assumptions for this repository.

Service Architecture

Service architecture defines the cloud resources, account information, cost controls, and production guardrails needed to turn this repo into a scoped service without publishing public financial assumptions.

Search And Service Surface

Public entry: free public benchmark/demo routes and sample traces
Paid boundary: paid hosted regression workspace, private benchmark scenarios, and provider routing dashboard
Canonical URL: https://stage-pilot.pages.dev/
Lead capture: https://kim3310-doeon-kim-portfolio.pages.dev/?offer=stage-pilot&inquiry=agent-reliability-audit#private-inquiry
Resource route: https://kim3310-doeon-kim-portfolio.pages.dev/resources/stage-pilot/
Commercial route: https://kim3310-doeon-kim-portfolio.pages.dev/?offer=stage-pilot#service-offers
Machine-readable offer: docs/service-offer.json
Search growth implementation: docs/search-growth-implementation.md
Revenue architecture: docs/revenue-architecture.md

Free Resource, Advertising, and Aggregate Data

Public utility and architecture checklist
Revenue model: contextual advertising on the policy-eligible central resource page.
Aggregate value: anonymous aggregate parser-recovery topic interest and guide-open counts
Boundary: ads allowed only on public reliability guide pages; traces, package diagnostics, and private scenario outputs are ad-free
Consent defaults off, DNT/GPC fail closed, and personal or sensitive data is never sold.

Name		Name	Last commit message	Last commit date
Latest commit History 106 Commits
.changeset		.changeset
.github		.github
.specify		.specify
.zed		.zed
docs		docs
examples		examples
experiments		experiments
infra		infra
scripts		scripts
site		site
src		src
tests		tests
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitleaks.toml		.gitleaks.toml
CHANGELOG.md		CHANGELOG.md
CLOUDFLARE_PAGES.md		CLOUDFLARE_PAGES.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
IMPLEMENT.md		IMPLEMENT.md
LICENSE		LICENSE
NOTICE.md		NOTICE.md
README.md		README.md
SECURITY.md		SECURITY.md
SUPPORT.md		SUPPORT.md
biome.jsonc		biome.jsonc
codecov.yml		codecov.yml
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.build.json		tsconfig.build.json
tsconfig.json		tsconfig.json
tsconfig.vitest.json		tsconfig.vitest.json
tsup.config.ts		tsup.config.ts
vercel.json		vercel.json
vitest.config.ts		vitest.config.ts
wrangler.toml		wrangler.toml

Navigation Menu

Repository files navigation

StagePilot

Provenance And Ownership

Three-Minute Proof

System Overview

Evaluation Path

Service Launch Playbook

Architecture Notes

The Problem

The Solution

Architecture

Stage-Gated Pipeline Detail

Benchmark Results

30 Mutation Modes

Quick Start

With The Upstream Middleware

As full runtime (API server)

Middleware Variants

API Endpoints

Deployment

Observability Stack

Project Layout

Architecture Decision Records

Tech Stack

Links

License

Cloud + AI Architecture

Enterprise Productization

System Architecture

Service Architecture

Search And Service Surface

Free Resource, Advertising, and Aggregate Data

About

Releases

Packages

Used by

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

StagePilot

Provenance And Ownership

Three-Minute Proof

System Overview

Evaluation Path

Service Launch Playbook

Architecture Notes

The Problem

The Solution

Architecture

Stage-Gated Pipeline Detail

Benchmark Results

30 Mutation Modes

Quick Start

With The Upstream Middleware

As full runtime (API server)

Middleware Variants

API Endpoints

Deployment

Observability Stack

Project Layout

Architecture Decision Records

Tech Stack

Links

License

Cloud + AI Architecture

Enterprise Productization

System Architecture

Service Architecture

Search And Service Surface

Free Resource, Advertising, and Aggregate Data

About

Topics

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages