Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,9 @@ _Avoid_: truncated (reserved for retention-ring capacity loss)
### Observation

**Snapshot**:
A point-in-time capture of a session's current screen, optionally with detected elements.
A point-in-time capture of a session's current screen text, cursor, and dimensions.
_Avoid_: frame, capture (as a noun)

**Element**:
A UI control detected on the screen (button, input, toggle) with its position and label.
_Avoid_: widget, component

**Revision**:
A monotonic per-session counter that advances whenever screen state may have changed;
the identity used for change detection and future diffs.
Expand Down
1 change: 0 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ anyhow = "1"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
regex = "1"
unicode-width = "0.2"

# System
libc = "0.2"
Expand Down
47 changes: 6 additions & 41 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,16 +210,9 @@ The `snapshot` command returns structured data about the terminal screen:
```json
{
"outcome": "immediate",
"snapshot_id": 42,
"size": { "cols": 80, "rows": 24 },
"cursor": { "row": 5, "col": 10, "visible": true },
"text": "Options: [x] Enable [ ] Debug\nActions: [OK] [Cancel]",
"elements": [
{ "kind": "toggle", "row": 0, "col": 9, "width": 3, "text": "[x]", "confidence": 1.0, "checked": true },
{ "kind": "toggle", "row": 0, "col": 22, "width": 3, "text": "[ ]", "confidence": 1.0, "checked": false },
{ "kind": "button", "row": 1, "col": 9, "width": 4, "text": "[OK]", "confidence": 0.8 },
{ "kind": "button", "row": 1, "col": 14, "width": 8, "text": "[Cancel]", "confidence": 0.8 }
],
"content_hash": 12345678901234567890
}
```
Expand All @@ -229,33 +222,6 @@ JSON snapshots always include an `outcome`: `immediate`, `changed`, `settled`,
instead of replacing it with an error. Exited captures include process exit metadata and
whether the final output was completely drained.

## UI Elements (Contextual)

pilotty automatically detects interactive UI elements in terminal applications. Elements provide **read-only context** to help understand UI structure, with position data (row, col) for use with the click command.

**Use keyboard navigation (`pilotty key Tab`, `pilotty key Enter`, `pilotty type "text"`) for reliable TUI interaction** rather than element-based actions, as UI element detection depends on visual patterns that may disappear after interaction.

### Element Kinds

| Kind | Detection Patterns | Confidence |
|------|-------------------|------------|
| **button** | Inverse video, `[OK]`, `<Cancel>` | 1.0 / 0.8 |
| **input** | Cursor position, `____` underscores | 1.0 / 0.6 |
| **toggle** | `[x]`, `[ ]`, `☑`, `☐` | 1.0 |

### Element Fields

| Field | Description |
|-------|-------------|
| `kind` | Element type: `button`, `input`, or `toggle` |
| `row` | Row position (0-based) |
| `col` | Column position (0-based) |
| `width` | Width in terminal cells |
| `text` | Text content of the element |
| `confidence` | Detection confidence (0.0-1.0) |
| `focused` | Whether element has focus (only present if true) |
| `checked` | Toggle state (only present for toggles) |

### Wait for Screen Changes

The `--await-change` flag solves the fundamental problem of TUI automation: **"How long should I wait after an action?"**
Expand Down Expand Up @@ -334,19 +300,18 @@ fi
### Workflow Example

```bash
# 1. Spawn a TUI with dialog elements
# 1. Spawn a dialog
pilotty spawn dialog --yesno "Continue?" 10 40

# 2. Wait for dialog to render
pilotty wait-for "Continue"

# 3. Get snapshot with elements (for context)
pilotty snapshot | jq '.elements'
# Shows detected buttons, helps understand UI structure
# 3. Inspect the visible screen
pilotty snapshot --format text

# 4. Navigate and interact with keyboard (reliable approach)
pilotty key Tab # Move to next element
pilotty key Enter # Activate selected element
# 4. Navigate and interact with keyboard
pilotty key Tab # Move to the next control
pilotty key Enter # Activate the selected control
```

## Sessions
Expand Down
2 changes: 1 addition & 1 deletion crates/pilotty-cli/src/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ const SESSION_HELP: &str = "Target session by name or ID [default: default]";
///
/// Spawn TUI applications in managed PTY sessions and interact with them
/// programmatically. Designed for AI agent consumption with structured
/// JSON output and stable element references.
/// JSON output and deterministic terminal input.
#[derive(Debug, Parser)]
#[command(name = "pilotty", version)]
pub struct Cli {
Expand Down
17 changes: 8 additions & 9 deletions crates/pilotty-cli/src/daemon/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@ mod tests {
}

#[tokio::test]
async fn shipped_v1_daemon_is_probed_and_current_command_is_rejected_before_transmission() {
async fn shipped_v1_daemon_is_probed_and_output_is_rejected_before_transmission() {
let socket_path =
std::env::temp_dir().join(format!("pilotty-probe-{}.sock", std::process::id()));
let _ = std::fs::remove_file(&socket_path);
Expand Down Expand Up @@ -355,21 +355,20 @@ mod tests {
stream,
daemon_protocol: None,
};
let command = Command::Output {
session: None,
ansi: false,
};
let required_protocol = command.minimum_protocol();
let error = client
.request(Request::new(
"output-request",
Command::Output {
session: None,
ansi: false,
},
))
.request(Request::new("output-request", command))
.await
.expect_err("old daemon must be rejected");

let message = error.to_string();
assert!(message.contains("speaks protocol 1"), "got: {message}");
assert!(
message.contains(&format!("requires protocol {PROTOCOL_VERSION}")),
message.contains(&format!("requires protocol {required_protocol}")),
"got: {message}"
);
peer.await.expect("protocol fixture task");
Expand Down
Loading
Loading