@@ -11,11 +11,11 @@ tags:
 draft: false
 ---
 
-Give an agent last week's conversation and one of two things usually happens. The session log is pasted into the prompt, or chunks go into a vector store and a handful of near neighbours come back. The first fills the window. The second has no path for which project or which user the memory belongs to.
+Give an agent last week's conversation and one of two things usually happens. The session log is pasted into the prompt, or chunks go into a vector store and a handful of near neighbours come back. The first fills the window. The second can use namespaces or metadata filters to restrict results by project and user, but it does not by itself give the agent a path-based walk from summaries to source files.
 
 [OpenViking](https://github.com/volcengine/OpenViking) puts that context on a `viking://` virtual filesystem. An agent uses `ls`, `tree`, `read`, and `write`, checks a directory summary for relevance, then opens the original file. It is open source from ByteDance's Volcano Engine Viking team. The main project is AGPLv3.
 
-This article is a reading of the retriever, filesystem service, and session extraction code at commit `0f77ab5` on 12 September 2026. It is not a measurement from a running server. LoCoMo and tau2-bench figures are vendor results from the project README and the [benchmark write-up](https://blog.openviking.ai/post/openviking-benchmark-results/).
+This article is a reading of the retriever, filesystem service, and session extraction code at commit [`0f77ab5`](https://github.com/volcengine/OpenViking/tree/0f77ab5) on 12 September 2026. It is not a measurement from a running server. LoCoMo and tau2-bench figures are vendor results from the project README and the [benchmark write-up](https://blog.openviking.ai/post/openviking-benchmark-results/).
 
 [Context engineering](/en/blog/30-context-engineering/) argued that a larger window still leaves the problem of what to load. OpenViking is an implementation that moves that choice onto paths and layers. It is a different layer from a company-wide brain ([Cerebras](/en/blog/37-company-brain/)) or vector search beside operational data ([TencentDB](/en/blog/50-tencentdb-agent-memory/), [DynamoDB](/en/blog/55-dynamodb-vector-search/)). The question here is **which directory, at which depth**, the agent reads.
 
 
@@ -1,5 +1,5 @@
 ---
-title: "DeepSeek Harness code analysis: swapping even the agent loop as a plugin"
+title: "DeepSeek Harness code analysis: making the agent loop a plugin boundary"
 summary: "DeepSeek Harness treats the model, tools, and session — and the loop that drives a turn — as plugins. Cordis mounts services on ctx keys; the default driver is ctx.agentLoop. This splits what is kernel from what you change in config, at the 10 September 2026 commit."
 date: "2026-09-13T10:20:00+09:00"
 tags:
@@ -15,7 +15,7 @@ A small change to a coding agent often turns into a fork of the whole harness. Y
 
 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`) is an open-source harness that splits those parts into plugins. The slogan is “Everything is a Plugin.” Model adapters, tools, sessions, sandboxes, scheduling, and the UI are plugins — and so is the **default agent loop**, the `dsh-agent-loop` package. The license is MIT. The docs call it a developer preview and warn that compatibility will break.
 
-This article reads `docs/architecture.md` and `packages/core` at commit `c291e79` (0.1.5 sync) on 10 September 2026. It is not a log from `npx @deepseek-ai/dsh web`. It takes the claim from [harness engineering](/en/blog/35-harness-engineering/) — you design the outside of the model — and checks it in a tree that makes the loop itself a replaceable unit. The same-day [OpenViking](/en/blog/63-openviking/) piece is about treating memory locations as paths.
+This article reads [`docs/architecture.md`](https://github.com/deepseek-ai/deepseek-harness/blob/c291e79/docs/architecture.md) and `packages/core` at commit [`c291e79`](https://github.com/deepseek-ai/deepseek-harness/tree/c291e79) (0.1.5 sync) on 10 September 2026. It is not a log from `npx @deepseek-ai/dsh web`. It takes the claim from [harness engineering](/en/blog/35-harness-engineering/) — you design the outside of the model — and checks it in a tree that makes the loop itself a replaceable unit. The same-day [OpenViking](/en/blog/63-openviking/) piece is about treating memory locations as paths.
 
 ## Separate the slogan from the kernel
 
 
@@ -0,0 +1,95 @@
+---
+title: "Claude Code plugin eval: compare the same task with and without a plugin"
+summary: "Claude Code 2.1.269 added plugin eval. It repeats the same prompts with and without a plugin and reports WITH, W/OUT, and delta scores. This article explains how to design cases and graders and use the results for regression testing."
+date: "2026-09-13T15:00:00+09:00"
+tags:
+  - claude-code
+  - plugin
+  - ai-eval
+  - agent-engineering
+draft: false
+---
+
+A Claude Code plugin packages skills, hooks, agents, and MCP servers. `claude plugin validate` can tell you whether that package is well formed. It cannot tell you whether the plugin improves the outcome of the same task. A polished description and one good demo are not enough when the model may skip a skill or a hook may intervene at the wrong time.
+
+Anthropic added `claude plugin eval` in the [Claude Code 2.1.269 release](https://github.com/anthropics/claude-code/releases/tag/v2.1.269) published on 11 September 2026. It runs a plugin's eval suite through Claude Code and produces JSON and HTML reports. The broader evaluation framework belongs in the [AI evaluation guide](/en/guides/ai-eval/); this article checks the CLI help in 2.1.270 to explain what this command compares and how to interpret it. It does not present measurements from a plugin run.
+
+## Remove only the plugin from the same task
+
+The default comparison holds the case prompt and graders fixed and changes whether the plugin is loaded.
+
+```text
+same case
+├── WITH    Claude Code run with the plugin loaded
+└── W/OUT   baseline run without the plugin
+       ↓
+     Δ = WITH - W/OUT
+```
+
+Read the report columns this way:
+
+| Field | Meaning | What to inspect |
+|---|---|---|
+| WITH | Aggregate score with the plugin | Whether the intended skill or tool ran and the result met the contract |
+| W/OUT | Baseline score on the same prompt without the plugin | Whether the model already solved the task unaided |
+| Δ | WITH minus W/OUT | Direction and size of the plugin's added effect |
+| RUNS | Repeated runs in each condition | How much evidence exists beyond one lucky completion |
+
+A positive delta does not establish that a plugin is good. Model runs vary, and a grader can reward the wrong proxy. A high WITH score adds little when W/OUT is equally high. A good mean delta should not hide a run that damaged a repository or used a forbidden tool.
+
+## Create cases and graders
+
+From the plugin root:
+
+```bash
+claude plugin eval init
+claude plugin eval .
+```
+
+By default, `init` starts an interview. Claude reads the plugin, asks which behavior to protect, and creates cases and graders under `evals/`. Use `claude plugin eval init --bare <name>` for a blank template. A case can be a single `case.yaml`, or a `prompt.md` accompanied by `graders/*.md`.
+
+A useful case does not tell Claude to use the plugin. It uses a realistic request and tests whether the plugin activates when needed. A code-review plugin, for example, needs three kinds of cases:
+
+1. **Should activate** — find a real defect in changed files and rank the fix.
+2. **Should stay out** — a small typo where the plugin's full procedure only adds cost.
+3. **Known failure** — a large change or a repository without tests that previously caused omissions.
+
+Attach graders to the work contract rather than the prose style of the final answer. Check required defects, allowed tools, exact file effects, and test results separately. Splitting deterministic file and tool checks from subjective model judging makes a score easier to diagnose.
+
+## More runs add evidence and cost
+
+The CLI defaults to three runs per case; `--runs <n>` overrides it. A with/without comparison therefore needs three WITH and three W/OUT agent sessions for one case, plus any paid grader calls. More runs expose variance, while also increasing usage and elapsed time.
+
+Three runs are useful while fixing obvious failures. Increase the count only for release candidates. Keep per-run scores and failure reasons, not just the mean. A plugin that succeeds three times out of five remains hard to trust even if its average looks good.
+
+Fix the model, permissions, tools, and fixture as well. Changing the model and plugin together makes the source of the delta unknowable. Record `--model`, allowed tools, case files, and grader versions with the result. MCP servers use recorded mocks by default; real servers start only when explicitly allowed.
+
+```bash
+claude plugin eval . \
+  --runs 5 \
+  --model sonnet \
+  --max-cost-usd 20 \
+  --no-publish \
+  --report evals/results/plugin-report.html
+```
+
+`--threshold` exits with code 1 when any case falls below the chosen score, which makes it usable as a CI regression gate. `--max-cost-usd` checks the ceiling before starting each run. In-flight parallel runs can still carry the total over the ceiling, so pair it with `--concurrency 1` for a strict budget.
+
+## Change plugins from observed failures
+
+The larger change is the order of plugin development. Instead of polishing a skill description and replaying a successful demo, capture a past failure as a case, measure WITH against W/OUT, and then edit the plugin.
+
+- If both arms score well and delta is small, the problem may not need a plugin.
+- If both arms score poorly, inspect the task, tools, and graders before the plugin.
+- If WITH is unstable, trace the plugin path: skill selection, hook timing, and MCP responses.
+- If delta improves while time and cost jump, limit the plugin to work where that quality gain pays for itself.
+
+Adoption criteria should therefore include success rate, elapsed time, usage cost, human intervention, and forbidden actions alongside score. Keeping representative tasks fixed also exposes regressions after model updates or plugin edits.
+
+## Check before running
+
+Eval runs launch Claude Code child processes locally with the signed-in user's permissions and usage. Review an unfamiliar plugin's eval suite as code. Scaffold scripts and real MCP servers run as the user when their explicit options are enabled. Passing cases do not constitute a security review.
+
+The HTML report may be published to claude.ai when account and policy gates allow it. Use `--no-publish` when repository material must stay local. Check `claude plugin eval --help` for the options and availability in the installed build.
+
+Plugin eval changes the question from whether a plugin looks useful to how much it changes the outcome of the same task. Three real failures that you do not want to repeat are a better starting suite than a large set of synthetic prompts. Once those failures become cases and graders, delta becomes evidence for improving the plugin rather than a line in its description.
@@ -11,11 +11,11 @@ tags:
 draft: false
 ---
 
-에이전트에게 지난 대화를 맡기면 흔히 두 가지 중 하나가 된다. 세션 로그를 프롬프트에 그대로 붙이거나, 벡터 DB에 넣고 비슷한 조각을 몇 개 꺼낸다. 전자는 토큰이 금방 차고, 후자는 어느 프로젝트·어느 사용자 기억인지 경로가 없다.
+에이전트에게 지난 대화를 맡기면 흔히 두 가지 중 하나가 된다. 세션 로그를 프롬프트에 그대로 붙이거나, 벡터 DB에 넣고 비슷한 조각을 몇 개 꺼낸다. 전자는 토큰이 금방 찬다. 후자도 namespace나 메타데이터 필터로 사용자·프로젝트 범위를 제한할 수 있지만, 에이전트가 그 경로를 따라 요약에서 원문으로 내려가는 탐색 절차까지 제공하지는 않는다.
 
 [OpenViking](https://github.com/volcengine/OpenViking)은 이 컨텍스트를 `viking://` 가상 파일 시스템으로 둔다. 에이전트는 `ls`, `tree`, `read`, `write`로 디렉터리를 보고, 디렉터리 요약으로 관련 여부만 확인한 뒤 원문을 연다. 바이트댄스 볼케이노 엔진 Viking 팀이 만든 오픈소스고, 본 프로젝트 라이선스는 AGPLv3다.
 
-이 글은 2026년 9월 12일 `0f77ab5` 커밋을 기준으로 저장소의 검색기·파일 서비스·세션 추출 코드를 읽은 분석이다. 서버를 띄워 재현한 측정은 아니다. LoCoMo·tau2-bench 숫자는 프로젝트 README와 [벤치마크 글](https://blog.openviking.ai/post/openviking-benchmark-results/)에 적힌 공급자 평가다.
+이 글은 2026년 9월 12일 [`0f77ab5`](https://github.com/volcengine/OpenViking/tree/0f77ab5) 커밋을 기준으로 저장소의 검색기·파일 서비스·세션 추출 코드를 읽은 분석이다. 서버를 띄워 재현한 측정은 아니다. LoCoMo·tau2-bench 숫자는 프로젝트 README와 [벤치마크 글](https://blog.openviking.ai/post/openviking-benchmark-results/)에 적힌 공급자 평가다.
 
 [컨텍스트 엔지니어링](/blog/30-context-engineering/)에서 창이 커져도 넣을 것을 고르는 일이 남는다고 썼다. OpenViking은 그 고르기를 경로와 계층으로 옮긴 구현이다. 회사 전체를 한 브레인으로 묶는 [Cerebras 사례](/blog/37-company-brain/)나 운영 데이터 옆 벡터 검색([TencentDB](/blog/50-tencentdb-agent-memory/), [DynamoDB](/blog/55-dynamodb-vector-search/))과는 층이 다르다. 여기서 보는 것은 에이전트가 어느 디렉터리의 어느 깊이를 읽을지다.
 
 
@@ -1,5 +1,5 @@
 ---
-title: "DeepSeek Harness 코드 분석: 에이전트 루프까지 플러그인으로 갈아끼우기"
+title: "DeepSeek Harness 코드 분석: 에이전트 루프를 플러그인 경계로 나눈 구조"
 summary: "DeepSeek Harness는 모델·도구·세션뿐 아니라 에이전트가 한 바퀴 도는 루프도 플러그인이다. Cordis가 서비스를 ctx 키로 붙이고, 기본 드라이버는 ctx.agentLoop다. 2026년 9월 10일 커밋을 기준으로 무엇이 커널이고 무엇을 설정으로 바꾸는지 나눈다."
 date: "2026-09-13T10:20:00+09:00"
 tags:
@@ -15,7 +15,7 @@ draft: false
 
 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(`dsh`)는 그 부품을 플러그인으로 나눈 오픈소스 실행기다. 슬로건은 “Everything is a Plugin”이다. 모델 어댑터, 도구, 세션, 샌드박스, 스케줄, UI뿐 아니라 **기본 에이전트 루프도** `dsh-agent-loop` 패키지다. MIT 라이선스고, 문서는 developer preview라 호환이 깨지는 변경이 있다고 못 박는다.
 
-이 글은 2026년 9월 10일 `c291e79`(0.1.5 동기화) 기준으로 `docs/architecture.md`와 `packages/core`를 읽은 분석이다. `npx @deepseek-ai/dsh web`으로 돌려 본 사용 기록은 아니다. [하네스 엔지니어링](/blog/35-harness-engineering/)에서 말한 “모델 바깥을 설계한다”를, 루프 자체를 교체 단위로 만든 구현에서 확인한다. 같은 날 [OpenViking](/blog/63-openviking/)은 기억 위치를 경로로 다루는 쪽이다.
+이 글은 2026년 9월 10일 [`c291e79`](https://github.com/deepseek-ai/deepseek-harness/tree/c291e79)(0.1.5 동기화) 기준으로 [`docs/architecture.md`](https://github.com/deepseek-ai/deepseek-harness/blob/c291e79/docs/architecture.md)와 `packages/core`를 읽은 분석이다. `npx @deepseek-ai/dsh web`으로 돌려 본 사용 기록은 아니다. [하네스 엔지니어링](/blog/35-harness-engineering/)에서 말한 “모델 바깥을 설계한다”를, 루프 자체를 교체 단위로 만든 구현에서 확인한다. 같은 날 [OpenViking](/blog/63-openviking/)은 기억 위치를 경로로 다루는 쪽이다.
 
 ## 슬로건과 커널을 나누어 본다