Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 46 additions & 12 deletions docs/AGENT_RUNTIME_V1.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ session.
- Persist background agent work as first-class jobs.
- Wake jobs from alarms, boot, package replacement, and explicit tool calls.
- Keep delivery, retries, diagnostics, and failure state with the job.
- Preserve the reviewed autonomy posture: background jobs do not silently
perform state-changing actions.
- Preserve the reviewed autonomy posture: background jobs pause before
state-changing actions and resume only from an exact Android-owned review.
- Keep model routing in the registry and schemas instead of Java keyword
routing.

Expand Down Expand Up @@ -45,18 +45,35 @@ An `agent_job` records durable background work.
"mode": "notification",
"notification_text": "Release notes changed"
},
"status": "active",
"status": "queued",
"created_at": 1781480000000,
"updated_at": 1781480000000,
"next_run_at": 1781483055000,
"running_at": 0,
"last_run_at": 0,
"last_result": "",
"failure_count": 0,
"failure_alert_at": 0
"failure_alert_at": 0,
"phase": "queued",
"progress_text": "Queued",
"progress_current": 0,
"progress_total": 0,
"checkpoint_json": {},
"pending_confirmation_id": "",
"pending_tool_request_json": {},
"last_surface_id": "",
"resume_token": "",
"last_event_at": 1781480000000,
"unread_result": false,
"paused_at": 0
}
```

Lifecycle states are `queued`, `running`, `waiting`, `awaiting_review`,
`paused`, `completed`, `failed`, and `stopped`. Readers continue accepting the
legacy `active` value during migration. Runtime dispatch may also persist
`dispatched` while an external runtime owns the next step.

Job types:

- `agent_turn`: run a bounded model/tool loop against a prompt.
Expand All @@ -77,7 +94,8 @@ model/tool trace and must not be dumped directly into the notification surface.

The assistant schedules a single alarm for the next active job. On wake it:

1. repairs stale `running` jobs,
1. expires pending reviews and repairs stale `running` or review-resolution
jobs,
2. marks due jobs `running`,
3. starts bounded background runners,
4. persists completion/failure,
Expand All @@ -89,13 +107,29 @@ service startup.

## Safety

Background jobs may call observe/read tools and terminal tools. State-changing
tools return `background.confirmation_required` instead of executing. The
foreground reviewed flow remains the only path for actions that mutate device,
account, communication, or external state.

This is intentionally conservative. The follow-up design can add reviewed
background approvals with a notification action and a resumable job checkpoint.
Background jobs may call observe/read tools and terminal tools directly. A
state-changing tool is never executed by the background model loop. Instead,
OpenPhone:

1. bounds and sanitizes the exact proposed tool arguments;
2. persists an `awaiting_review` checkpoint and 15-minute confirmation;
3. shows the exact action in an Android notification and AI Home run detail;
4. atomically claims the first Approve or Deny tap;
5. verifies the tool, parameter digest, runtime, phone session, job, expiry,
idempotency key, checkpoint, and resume token;
6. executes only the persisted request after approval, or records a structured
denial/timeout result;
7. resumes the job with that bound result instead of regenerating the action.

Provider secrets, raw authorization, screenshots, data URLs, large encoded
blobs, and unbounded context are rejected from checkpoints. Audit events store
digests and lifecycle metadata, not raw parameters. If the process dies while
an approved action is resolving, stale-run repair marks the outcome unknown
and does not replay it. A denied action can safely resume after restart.

The wire shapes are defined by `schemas/agent-job.schema.json` and
`schemas/background-confirmation.schema.json`; recorded valid, tampered, and
secret-bearing fixtures run in `./scripts/check.sh`.

## OpenClaw/Hermes Mapping

Expand Down
61 changes: 55 additions & 6 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,17 @@ Device
The current repo implements the first OpenPhone product layer:

- `vendor/openphone` product config.
- `OpenPhoneAssistant` privileged app with a chat-style user surface, text and
voice task entry, stateful mic/send/stop composer action, advanced model and
developer controls, task grants, screen context, trace/audit export, OTA
preview controls, and audit controls.
- `OpenPhoneAssistant` privileged app with a dedicated Android Home activity,
a calm voice-first AI Home surface, explicit conventional-launcher App Space
handoff, text and voice task entry, stateful mic/send/stop controls, advanced
model and developer controls, task grants, screen context, trace/audit
export, OTA preview controls, and audit controls. AI Home is an immersive
normal application window rather than an opaque overlay: it hides Android
status/navigation chrome while foreground and allows transient system bars
by edge swipe, while Android retains ownership of secure keyguard,
notifications, IME, recents, emergency UI, and system dialogs. OpenPhone
disables only the non-secure swipe keyguard by default; configuring a
pattern, PIN, or password restores Android's credential gate.
- Initial capability and policy config files. `scripts/check.sh` validates that
the assistant fallback `PolicyEngine` covers every capability in
`openphone_capabilities.json` with the same risk class.
Expand All @@ -66,9 +73,51 @@ The current repo implements the first OpenPhone product layer:
- Agent Runtime V1 background jobs. The first assistant-side implementation
persists background agent turns, schedules them with AlarmManager, wakes them
at boot/package replacement/service startup, delivers notification results,
and blocks state-changing background tools until a foreground reviewed
approval flow exists. The runtime contract is documented in
and pauses state-changing tools behind exact, expiring Android-owned review.
Approval or denial produces a bound checkpoint result; resume cannot
regenerate or substitute the authorized request, duplicate taps cannot
execute twice, and interrupted approval execution fails safe without replay.
The runtime contract is documented in
[AGENT_RUNTIME_V1.md](AGENT_RUNTIME_V1.md).
- A unified agent-run projection reads jobs, watchers, commitments, and
foreground execution sessions without creating a second execution store.
AI Home uses it for stable activity bubbles and run detail, while the
SystemUI island reads the same projection for compact recent/live status.
Approval-needed bubbles expose exact review detail and Approve/Deny actions;
queued work can be paused/resumed, and the compact island routes review back
to AI Home.
Inspection and dismissal state is local presentation metadata; source stores
remain authoritative for execution and stop operations.
- The compact OpenPhone island is owned by SystemUI. The assistant publishes a
bounded, privacy-minimized state projection through the OpenPhone agent
Binder service; `system_server` validates it, assigns revision/timestamps,
retains the latest snapshot across assistant UI recreation, and sends
one-way change events to SystemUI. SystemUI renders only a fixed-size
`TYPE_STATUS_BAR_SUB_PANEL` window with non-modal touch behavior, redacts
personal/review state on keyguard, routes detail and approval to AI Home, and
degrades to a generic bounded status when the publisher becomes stale. Its
compact geometry, black capsule, centered typography, and terse state glyphs
preserve the established assistant-island presentation while moving window
ownership into SystemUI. The
assistant's full-screen `TYPE_SYSTEM_ERROR` layer remains only for
non-touchable pointer/glow visualization during active device control; the
old assistant island is a property-gated compatibility renderer.
- Adaptive Surface V1 is a phone-owned, revisioned UI document contract.
Runtimes can request only a bounded semantic component registry; the
assistant validates identity, ownership, expiry, depth/node/text/image
limits, local artifact provenance, registered tool bindings, tool parameter
shapes, accessibility labels, and mutation disclosure before persistence or
rendering. Surface actions resolve against the current revision and then
enter the existing runtime tool bridge, so policy, grants, confirmation,
idempotency, and framework audit remain authoritative. The Compose renderer
has no WebView, HTML, remote image URL, arbitrary color, absolute-position,
or runtime-code path.
- Built-in calendar search and message/notification result paths produce
deterministic Adaptive Surface documents from trusted phone tool results.
Tool artifacts retain session/runtime/capability/sensitivity/expiry
provenance separately from the layout, while the surface repository survives
activity recreation and fails closed on stale, malformed, expired, or
cross-session content.
- Runtime Agent Protocol for external agent runtimes. This is an OpenPhone AI
layer boundary, not a separate product stack: Android owns sessions, screen
context, tool execution, confirmations, and audit, while adapters map remote
Expand Down
139 changes: 139 additions & 0 deletions docs/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,145 @@ Manual or screenshot-backed checks:
- tapping outside the composer dismisses the keyboard;
- recent logcat has no `FATAL EXCEPTION` or `AndroidRuntime` crash signature.

## AI Home And App Space Smoke Test

AI Home changes affect Android Home resolution and require a full product build
or an assistant APK plus the matching Launcher3 patch. An assistant-only APK
push does not remove Launcher3 as a competing Home candidate.

On a build containing both sides of the change, verify:

- after a reboot, the direct-boot-aware assistant service reports locked and
defers its island/runtime state while an early `OpenPhoneHomeActivity`
launch renders only its black locked-boot shell; neither path opens
credential-encrypted stores, and both initialize after `USER_UNLOCKED`
without crashing `org.openphone.assistant`;
- clean boot/setup resolves Home to
`org.openphone.assistant/.OpenPhoneHomeActivity`;
- an unsecured owner boots/wakes directly into AI Home without the
non-secure swipe keyguard, while a configured pattern, PIN, or password
continues to show Android's secure keyguard;
- the Home surface owns the full display without status or navigation chrome,
remains usable when no model runtime is configured, and exposes transient
system bars only after an edge swipe;
- holding the orb starts voice capture and release submits it;
- a short tap opens text entry;
- the visible Apps action and a two-finger inward pinch open the explicit
Launcher3 App Space activity;
- Launcher3 initializes Quickstep without a
`OverviewComponentObserver` null-home crash when App Space opens;
- pressing Home from App Space or another app returns to AI Home;
- tapping the compact idle island returns to AI Home;
- queued/running jobs, watchers, commitments, and foreground sessions appear
as stable Home activity bubbles without relaunching Home;
- Home shows no more than three individual activity bubbles plus a `+N`
overflow control, and the compact SystemUI island reports the same live-run
count and attention state;
- tapping a bubble shows its title, source kind, phase, latest progress, and
applicable Pause/Resume/Stop/Dismiss actions;
- stopping supported work updates its source store and survives assistant
process restart; dismissing a terminal result hides only its presentation
and never stops live work;
- terminal results remain visible until inspected and then age out after 24
hours; attention-required and live work sort ahead of recent terminal work;
- the notification shade, Quick Settings, secure keyguard, IME, recents,
system dialogs, camera, dialer, and emergency surfaces are not covered by a
full-screen OpenPhone overlay;
- TalkBack exposes the voice orb, Apps action, settings/history action, text
composer, review actions, run bubbles, overflow control, and run actions.

### SystemUI island checks

Run `node scripts/validate-island-contract.mjs` (also included in
`./scripts/check.sh`). The idle and approval-needed snapshots must pass, while
unknown-mode, secret-bearing, and oversized snapshots must fail. This validator
also checks the framework/SystemUI patch contract: a bounded status-bar
sub-panel, non-modal touch flags, keyguard redaction, stale-state degradation,
and no confirmation execution from the island itself.

This path requires a full product build containing framework patches `0020`
and `0021`; an assistant-only APK does not add the Binder listener or SystemUI
renderer. On a matching device, verify:

- idle, listening, working, live-run count, completion, error, and
approval-needed transitions update without recreating SystemUI;
- recreating or killing only the assistant activity leaves the latest
`system_server` snapshot rendered, while a publisher that stays absent beyond
the stale bound becomes a generic `OpenPhone · Offline` chip;
- opening AI Home hides the island and returning to App Space restores it;
- tapping the island dismisses keyguard when necessary and opens AI Home;
- long-press stops only an unlocked active foreground task; approval-needed,
stale, background-only, and locked states route to AI Home instead;
- approval details and personal status are replaced with a generic locked glyph
on keyguard, and Approve/Deny remain available only on AI Home;
- the App Space chip preserves the legacy compact 288 x 90 px proportions on
the Pixel 9a reference density, with a black capsule and terse state glyphs;
- touches immediately outside the fixed chip reach the underlying app, and
notification shade, Quick Settings, IME, recents, camera, dialer, and
emergency UI remain unobstructed;
- active pointer/glow visualization is non-touchable and disappears when
device control finishes.

### Resumable background review checks

Run `node scripts/validate-background-review-contract.mjs` (also included in
`./scripts/check.sh`). The known-valid `awaiting_review` job must pass, while a
modified-parameter request and a secret-bearing request must fail.

On a device, create a background job that proposes a registered state-changing
tool, then verify:

- the tool does not execute before review and the job becomes
`awaiting_review`;
- its Home bubble and notification show the exact tool and arguments;
- Deny returns `background.action_denied`, queues the same job, and lets it
complete gracefully;
- Approve executes only the persisted request and resumes from its stored tool
result;
- tapping Approve or Deny twice executes at most once;
- force-stopping and restarting the assistant before review preserves the
request and expiry;
- letting the request expire queues a structured timeout continuation and
removes the review notification;
- killing the assistant after an approval claim never replays an outcome whose
completion is unknown;
- audit/context events include binding digests and lifecycle state but no raw
tool parameters.

### Adaptive Surface V1 checks

Run `node scripts/validate-surface-contract.mjs` (also included in
`./scripts/check.sh`) and verify the known-valid calendar/message fixtures pass
while external-image, unknown-component, and unknown-action fixtures fail.

On a device, ask for a calendar list, message summary, and notification
summary, then verify:

- a deterministic surface replaces prose-only result presentation on AI Home;
- rotating/recreating Home restores the same surface and revision;
- dismissing it survives recreation, and an expired surface is not shown;
- every interactive element has a useful TalkBack label;
- action taps use the registered phone tool, and a stale revision is rejected;
- read-only actions execute through the tool bridge; mutating actions show the
existing local approval UI and cannot execute before approval;
- sensitive message/notification surfaces do not appear over a locked device;
- malformed documents, undeclared actions, arbitrary component types, and
remote image URLs render nothing and create a rejection event.

Useful resolution checks:

```bash
adb shell cmd package resolve-activity \
-a android.intent.action.MAIN \
-c android.intent.category.HOME

adb shell am start -W \
-n org.openphone.assistant/.OpenPhoneHomeActivity

adb shell am start -W \
-n com.android.launcher3/com.android.launcher3.uioverrides.QuickstepLauncher
```

If the mounted APK bytes match the new OTA but PackageManager still reports an
older persistent system-app version, treat it as stale `/data/system` package
metadata. On the Pixel 9a test device this happened after a v54 OTA: the
Expand Down
Loading
Loading