Release 0.19.0 - #398
Merged
Merged
Conversation
* feat(osmo): VS Code/Cursor dev workflow on NVIDIA OSMO
Adds a privileged Docker-in-Docker workspace task that lets a developer
run the full AirStack docker-compose stack on OSMO and attach an IDE
over SSH, with Isaac Sim WebRTC livestream + Foxglove websocket exposed
via osmo port-forward.
Components:
- osmo/workspace/{Dockerfile,entrypoint.sh,sshd_config}: airstack-osmo-workspace
image. Ubuntu 24.04 + sshd (pubkey-only) + Docker CE + Docker Compose +
nvidia-container-toolkit + fuse-overlayfs (DinD-on-overlayfs needs it,
otherwise dockerd falls back to vfs which bloats AirStack images ~10x).
- osmo/workflows/airstack-dev.yaml: single privileged GPU task. Materializes
Nucleus + airlab-docker secrets from OSMO credentials, clones AirStack,
starts inner dockerd, runs `airstack up` with desktop + isaac-sim-livestream
Compose profiles.
- simulation/isaac-sim: isaac-sim-livestream Compose service that runs
Pegasus standalone with --/app/livestream/enabled=true and exposes
WebRTC port ranges 47995-48012 / 49000-49007 / 49100; launch script
gates headless+livestream extension on ISAAC_SIM_LIVESTREAM env var.
- .airstack/modules/osmo.sh: airstack osmo:{up,ide,foxglove,webrtc,logs,down}
CLI wrappers around `osmo workflow submit` / `port-forward` / `cancel`.
Persists the active workflow id and validates it's still running before
each command (prevents the stale-state 410 error).
- airstack.sh: bash 4+ re-exec bootstrap (macOS ships 3.2; the CLI uses
`declare -A`).
- osmo/README.md + docs/tutorials/airstack_on_osmo.md: admin pool setup
(privileged_allowed) + per-user credentials (airlab-docker-login,
airlab-nucleus) + student-facing IDE attach + WebRTC/Foxglove flow.
Pool requirements: privileged_allowed: true, GPU pool with
nvidia-container-toolkit on the host, ample node ephemeral storage
(AirStack images extracted are ~50-100Gi via fuse-overlayfs; vfs needs
~500Gi+).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osmo): harden CLI + workspace image against stale-state, port-forward race, and cursor-server install hangs
Four bugs that bit the first end-to-end runs (airstack-dev-10 → -13):
- _osmo_wf_id: validate saved workflow id against `osmo workflow query`
before returning. Without this, the state file at ~/.airstack/osmo-state
outlives the workflow it points at and every subsequent osmo:webrtc /
osmo:foxglove / osmo:ide call surfaces the same confusing
"Workflow airstack-dev-N is not running! (status 410)" instead of the
obvious "run airstack osmo:up to launch a fresh workflow".
- cmd_osmo_up: `osmo workflow submit --set-env` is variadic. Passing two
separate `--set-env A=1 --set-env B=2` silently drops the first one —
this is what made airstack-dev-11 fail with "ERROR: SSH_PUB_KEY not set"
when --branch was passed alongside the pubkey. Collapse the K=V pairs
into a single --set-env.
- cmd_osmo_ide: previously launched the IDE before starting the
port-forward, so Cursor/VS Code would try to SSH localhost:2200 a few
hundred ms before the tunnel listener existed and fail with
"connect to host localhost port 2200: Connection refused". Now: detect
an existing forward and reuse it (also avoids the "Address already in
use" if osmo:foxglove was started in parallel), otherwise spawn the
forward in the background, wait up to 30s for it to bind, then launch
the IDE. Ctrl+C tears down the spawned forward cleanly via a trap.
- workspace image / entrypoint: Cursor Remote-SSH hung indefinitely
on airstack-dev-13 because (a) cursor-server's installer fell back to
wget when curl timed out and wget was not in the image, and (b) a
/tmp/cursor-remote-lock.* file left behind by the first crashed
install blocked every silent retry. Add wget to the apt install list
and rm -f the stale Cursor / VS Code remote lock files at the very
top of entrypoint.sh so each fresh pod starts from a clean slate.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osmo): correct osmo:logs CLI invocation; install Foxglove extensions locally on osmo:foxglove
osmo:logs was invoking `osmo workflow logs <id> workspace --follow`, but
the real CLI takes the task via `-t TASK` (not positionally) and has no
`--follow` flag at all — so the command failed immediately with
"unrecognized arguments: workspace --follow". Replace with a polling loop
that uses `-t workspace -n <N>` on a short interval, prints only the
suffix that appeared since the previous fetch (find-the-last-seen-line
trick; degrades to "reprint tail" with a warning if the cursor outruns
-n), and exits cleanly once the workflow reaches a terminal state.
Tunables: OSMO_LOGS_TASK / OSMO_LOGS_TAIL / OSMO_LOGS_INTERVAL.
osmo:foxglove now installs the AirStack Foxglove extensions
(robot-commands / waypoint-editor / polygon-editor) into the laptop's
local Foxglove user-extensions directory before opening the
port-forward. Without this, custom panels show up as "Unknown panel
type: robot-commands.Robot Tasks" in the laptop's Foxglove Desktop
because it has no way to discover the extension folders that live
inside the GCS container. To avoid duplicating the install logic, the
existing gcs/foxglove_extensions/install.py is refactored to read
FOXGLOVE_EXT_SRC / FOXGLOVE_EXT_DST env vars (the in-container call
already in gcs/docker/gcs-base-docker-compose.yaml keeps working
unchanged via defaults). The wrapper sets those vars to
${PROJECT_ROOT}/gcs/foxglove_extensions and
~/.foxglove-studio/extensions respectively, overridable with
OSMO_FOXGLOVE_EXT_DIR / skippable with OSMO_FOXGLOVE_SKIP_EXTENSIONS=1.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osmo): pin Kit livestream UDP media port to 49099 so osmo:webrtc actually shows pixels
Kit 107's WebRTC livestream picks a UDP media port dynamically. The
documented `omni.services.livestream.nvcf` defaults (minHostPort=47998
maxHostPort=48020 fixedHostPort=0) are ignored by the stock standalone
Kit binary — on airstack-dev-13 it bound to UDP 49042, outside both the
Compose-published range AND the default `osmo:webrtc --udp` forward of
`47995-48012,49000-49007`. Result: TCP signaling on 49100 worked, the
WebRTC Streaming Client window opened, but every SRTP media packet was
dropped → black viewport plus the recurring
`NVST_CCE_DISCONNECTED when m_connectionCount 0 != 1` underflow in Kit's log.
Pin the media port via three `app.livestream.*` settings set on
`SimulationApp` before `omni.kit.livestream.webrtc` is enabled, so
whichever code path the carb.livestream-rtc.plugin consults lands on the
same port:
app.livestream.fixedHostPort = 49099
app.livestream.minHostPort = 49099
app.livestream.maxHostPort = 49099
49099 is a deliberate one-off from the 49100 TCP signaling port — same
neighborhood, easy to remember. Verified live on airstack-dev-13 after
`docker compose up -d --force-recreate isaac-sim-livestream`: Kit binds
UDP 49099 (`/proc/net/udp` hex BFCB on 0.0.0.0) and docker-proxy
publishes it from the pod host network.
Knock-on cleanups:
- `simulation/isaac-sim/docker/docker-compose.yaml` shrinks the
isaac-sim-livestream `ports:` from 27 forwarded ports
(`47995-48012, 49000-49007 TCP+UDP, 49100 TCP`) to just two:
`49100/tcp` + `49099/udp`.
- `.airstack/modules/osmo.sh` shrinks `OSMO_WEBRTC_TCP` to `49100` and
`OSMO_WEBRTC_UDP` to `49099`, so `airstack osmo:webrtc` spawns two
port-forwards instead of thirty.
- `.gitignore` ignores `.DS_Store` so working from a Mac doesn't leak
Finder metadata.
After pulling this commit into a running pod: `docker compose up -d
--force-recreate isaac-sim-livestream` to apply the new port mapping;
then re-run `airstack osmo:webrtc` on the laptop to pick up the new
forward ranges. The standalone WebRTC Streaming Client connects to
`localhost` (same address as before) and now actually receives frames.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osmo): render Kit GUI in WebRTC stream; document SSH agent forward for in-pod git push
Two paper-cuts that bit airstack-dev-13 after the WebRTC media port pin
landed (commit 2d9b161):
(1) The WebRTC stream showed only the bare 3D viewport — no menu bar,
no toolbar, no panels, no console. Cause: SimulationApp's default
when `headless=True` is to also hide the UI (`hide_ui=True`). The
NVIDIA reference at
`simulation/isaac-sim/standalone_examples/api/isaacsim.simulation_app/livestream.py`
explicitly opts back into UI rendering plus picks explicit window
sizing and `display_options=3286` to keep the default grid/axes
visible. Mirror that config in `example_one_px4_pegasus_launch_script.py`
when `ISAAC_SIM_LIVESTREAM=true` (local desktop dev keeps the
minimal `headless=False` path unchanged).
(2) The pod has no SSH private key, only an `authorized_keys` for
inbound connections from the user's laptop. As a result, `git push`
from inside the Cursor / VS Code Remote-SSH session inside the pod
fails with "Permission denied (publickey)". sshd inside the
workspace image already has `AllowAgentForwarding yes` baked in via
`osmo/workspace/sshd_config`; the missing piece is purely on the
Mac side. Update the `~/.ssh/config` block in the tutorial to
include `ForwardAgent yes` (so the local agent's keys are exposed
in the pod), `AddKeysToAgent yes` (auto-load on first push), and
`UseKeychain yes` (macOS-only Keychain unlock without passphrase
prompts; ignored on Linux). Adds an `ssh-add -l` smoke-test note.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osmo): make osmo:setup idempotent + paste-safe; document Nucleus auth-debug path
osmo:setup hit two failure modes that wasted a debug session each:
- `osmo credential set` is not an upsert for GENERIC creds — re-running
setup (e.g. to rotate a Nucleus API token) failed with `400 duplicate
key value violates unique constraint "credential_pkey"` and bailed
before reaching the airlab-nucleus credential. Delete-then-set each
credential so re-running is idempotent.
- Bracket-paste mode and cross-OS clipboards routinely smuggle invisible
bytes around long pastes. Nucleus's auth endpoint silently DENIES a
token with one extra trailing byte, with no actionable error from the
client side. _osmo_prompt now strips leading/trailing whitespace and
CR/NUL bytes via a new _osmo_trim helper, and warns when bytes were
stripped. cmd_osmo_setup additionally JWT-shape-checks the Nucleus
token (must be eyJ.<dot>.<dot>.) before submitting it, so a wrong
paste fails at setup time instead of silently DENIED at pod boot.
Also documents how to debug the "Login Required: Unable to connect
server omniverse://airlab-nucleus..." popup: SSH the Nucleus host and
tail base_stack-nucleus-auth-1 for InternalCredentials.auth status:
DENIED. Adds a "Nucleus connectivity from OSMO" section to the admin
README clarifying that Nucleus over HTTPS uses a single 443 (no need
to open the native 3009-3180 range from the OSMO cluster), per
NVIDIA's TLS docs.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osmo): use Nucleus API-token auth, with double-dollar to survive compose parser
The OSMO entrypoint was writing OMNI_USER=<andrew_id> alongside an API
token JWT in OMNI_PASS, which routes the JWT through the password-
verification path. Nucleus silently DENIES — visible only in
base_stack-nucleus-auth-1 as `InternalCredentials.auth … 'username':
'<andrew>' … status: DENIED` (no Tokens.auth_with_api_token call). Kit
then pops "Login Required: Unable to connect server omniverse://...".
omniclient expects the literal sentinel username `$omni-api-token` paired
with the JWT as the password. The entrypoint now detects a JWT-shaped
OMNI_PASS (header starts with `eyJ`) and emits OMNI_USER=$$omni-api-token
into omni_pass.env. The `$$` is intentional: docker-compose v2
interpolates env_file values, and a single `$` would be eaten by the
parser (`OMNI_USER=$omni-api-token` becomes `OMNI_USER=-api-token` after
${omni}- expansion to empty). The container ultimately sees
OMNI_USER=$omni-api-token, which is the correct sentinel.
Also note for the next debugger: `docker compose restart` does NOT
re-read env_file. Use `docker compose up -d <svc>` to recreate the
container after editing omni_pass.env.
Updates omni_pass_TEMPLATE.env header to document the API-token pattern
explicitly (with the $$ caveat), and adds a troubleshooting row that
distinguishes "wrong auth path" (DENIED with no Tokens.auth_with_api_token
call) from "bad/expired token" (Tokens.auth_with_api_token: DENIED).
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(osmo): make OSMO the recommended dev path, single clone-the-repo flow
Reposition the OSMO tutorial as AirStack's recommended day-to-day
development path (not just a fallback for laptops without GPUs) and
collapse it onto a single recipe: clone the repo, then drive everything
through the airstack osmo:* wrappers in .airstack/modules/osmo.sh.
- docs/tutorials/airstack_on_osmo.md
- Retitle + rewrite the intro to lead with five concrete advantages
(pooled GPUs, no local CUDA/Docker/driver maintenance, same image as
CI + field robots, one-command onboarding, hardware bigger than your
laptop). Demote the Linux+GPU-desktop path to an escape hatch.
- Drop the Mac/Windows/no-GPU framing in 'Who is this for?' and the
mermaid laptop subgraph label.
- Add 'a local clone of AirStack' to Prerequisites; remove it from the
'do not need' list.
- Replace Option A/B credential split with a single
./airstack.sh osmo:setup recipe; move the three raw osmo credential
set calls into a collapsible 'Under the hood' footnote.
- Replace each step's raw osmo workflow ... command with the
corresponding airstack osmo:up/logs/ide/webrtc/foxglove/down wrapper;
preserve the raw form in 'Under the hood' footnotes that cross-link
cmd_osmo_* in .airstack/modules/osmo.sh.
- Drop the export WF=... paragraph — the wrappers read the id from
~/.airstack/osmo-state automatically; AIRSTACK_OSMO_WF overrides
per-invocation. \$WF now only appears inside the raw-form footnotes.
- Sweep Troubleshooting + What-survives tables: redirect raw
port-forward fixes to the airstack osmo:* equivalents and rename the
section to 'What survives airstack osmo:down?'.
- Fix WebRTC edge label (49100/tcp + 49099/udp) to match the pinned
ports the workflow actually uses today.
Companion cleanups now that the privileged_allowed flip is automatic on
the OSMO autosync side (synchronize_osmo_team_pools.py forces
privileged_allowed: true on every platform of every pool, so students
never see the 'platform does not have privileged flag enabled' error):
- osmo/README.md: drop the 'Most common blocker' privileged warning, the
privileged_allowed row from the pool-requirements table, and the
'privileged GPU pod' / '(privileged, GPU)' descriptors in the
architecture summary. Simplify the validation-stage SSH-failure hint.
- osmo/workflows/airstack-dev.yaml: trim the long DinD-requires-privileged
comment to a one-liner (the privileged: true directive itself stays).
- .airstack/modules/osmo.sh: remove the special-case 'privileged flag
enabled' error branch in cmd_osmo_up — it should never fire now.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osmo): make osmo:logs actually stream + survive pod host-key churn
osmo:logs was silent because cmd_osmo_logs wrapped osmo workflow logs in
$( ... ) on the assumption that -n LAST_N_LINES exits after dumping the
tail. Empirically the CLI keeps the stream open as new lines arrive (it
already behaves like tail -f, despite --help advertising only -n), so
command substitution waited forever and printed nothing. Drop the polling
loop and just exec the command directly.
Each fresh OSMO pod also ships a new sshd host key, so every osmo:up
trips StrictHostKeyChecking against the previous workflow's fingerprint
and SSH/Cursor abort with "Host key for [localhost]:2200 has changed".
Switch the recommended ~/.ssh/config block (and osmo/README.md) to the
ephemeral-host pattern (StrictHostKeyChecking no + UserKnownHostsFile
/dev/null + LogLevel ERROR), and have cmd_osmo_ide ssh-keygen -R the
stale loopback entry on every run so users on the old config get
unblocked automatically.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osmo): auto-pin --branch to local checkout + clean error UX when workflow dies
The pod's entrypoint clones AirStack fresh from GitHub on every workflow
start (the pod fs is ephemeral). It defaulted to `main`, so any developer
testing branch-only OSMO changes silently ran their pod against stale
`main` code — most visibly: COMPOSE_PROFILES=desktop,isaac-sim-livestream
resolved to "desktop" alone on `main` because the isaac-sim-livestream
service only exists on the feature branch, so isaac-sim never came up
and `airstack osmo:webrtc` showed a blank stream.
- cmd_osmo_up now defaults --branch to the local repo's current
branch (git rev-parse --abbrev-ref HEAD). Detached HEAD or
non-git checkouts fall back to `main` cleanly. Pass --branch
explicitly to override.
- New _osmo_check_branch_pushed warns up-front when the about-to-
submit branch has no upstream, is ahead of origin, or has an
uncommitted working tree. The pod doesn't see your laptop's edits.
Separately, when an OSMO workflow gets canceled mid-flight (osmo:down
in another shell, or OSMO timing it out), the in-flight port-forward
and logs streams raise OSMOUserError("Workflow X is not running!")
from inside an asyncio Task. The CLI prints "Task exception was never
retrieved" + a multi-line Traceback that buries the actual one-line
cause. New _osmo_pf_filter awk script collapses that into a single
[ERROR] line pointing at `airstack osmo:up`. Wired into webrtc,
foxglove, and logs. webrtc also gains a cleanup trap that kills the
backgrounded UDP port-forward on EXIT/INT/TERM so we don't leak it
against a dead workflow.
Tutorial Step 2 documents the new --branch default and the
"pod-clones-from-GitHub-not-your-laptop" gotcha.
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(osmo): bump inner dockerd concurrency to saturate 10 GbE pulls
dockerd's defaults of --max-concurrent-downloads=3 / --max-concurrent
-uploads=5 cap a fresh airstack-dev pod's image-pull at ~300 MiB/s
against the airlab-backup-10g registry — single-stream TLS tops out
around 300-500 MiB/s per core, and three parallel streams of unevenly
sized blobs serialize down to that ceiling. Ceph (1014 TiB, 92 OSDs,
SSD pools) and 10 GbE both have far more headroom than that. Bump to
10/10 to overlap enough blob downloads to saturate the pipe.
Threaded through the DOCKERD_MAX_DOWNLOADS / DOCKERD_MAX_UPLOADS env
vars so a pool can be tuned at submit time without rebuilding the
workspace image.
Workspace image needs a rebuild + push for this to take effect:
cd osmo/workspace
docker build -t airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest .
docker push airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(osmo): require buildx --platform linux/amd64 for workspace image
A plain `docker build && docker push` on an Apple Silicon Mac silently
produces a linux/arm64-only `latest` manifest. OSMO workers are amd64,
so every subsequent workflow fails at the outer pod-image pull with
"no match for platform in manifest" before the entrypoint even runs —
a confusing failure mode whose root cause lives entirely in the push,
not in the workflow yaml or the entrypoint.
Switch the README and the Dockerfile docstring to the buildx form,
explain the why, and document the post-push manifest check.
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(osmo): move dockerd data-root to /osmo/run for native overlay2
The OSMO pod's `/` is itself a containerd overlay snapshot, and Linux
refuses to stack a second overlayfs on top of an overlay rootfs — which
is why the inner dockerd was falling through to fuse-overlayfs. That
costs a kernel↔userspace FUSE round-trip on every `creat()` during
layer extraction, which murders throughput on apt/pip/ROS layers
(measured: 32-50 MB/s for small-file-heavy layers vs 480 MB/s for
big-file layers in the same pull).
Pointing dockerd at /osmo/run/docker (the kubelet emptyDir backed by
ext4 on /dev/vda3) lets the existing overlay2-first fallback chain
actually succeed on its first try, restoring kernel-overlay extraction
performance. emptyDir lifetime matches the workflow lifetime, so the
docker layer cache gets the right scope automatically.
Falls back to /var/lib/docker if /osmo/run isn't present so the image
still works in non-OSMO test contexts.
Co-authored-by: Cursor <cursoragent@cursor.com>
* updated version
* added virtual display for GL context
* added virtual display for droan_gl
* droan_gl patch
* run Xvfb in its own tmux session
* updated dockerfile + version
* typo in docs
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* typo in comments
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* typo in comments
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* typo in comments
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* typo in osmo logs, renamed airstack-isaac-sim to just isaac-sim
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* typo in container name for isaac-sim-livestream
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* airstack-dev version overwrite removed
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: krrishj18 <krrishj@andrew.cmu.edu>
Co-authored-by: Andrew Jong <ajong@andrew.cmu.edu>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update submodule to point to pegasus fix fixing start/stop behavior * Bump VERSION to 0.19.0-alpha.2 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* incremented version tag * docker image builds on l4t with generalizability features for other ros and linux versions * documentation and claude skills for developing a new profile. * initial natnet implementation * deployment to jetson with ros2 jazzy now fixed * unit testing dependency fix * added optitrack perception to launch * put tag version back in * added instructions for Agents to run tests * attempt at completely custom Optitrack Parser (Not working) * fully implemented NatNetSDK natnet ros2 wrapper natively in AirStack. Hand test in mocap room successful * unit test restructuring * reorganized natnet logic for unit-testability * unit testing restructuring to have unit tests in src and proxies in test. Unit tests workflows created * reupdated documentation for current state of testing * change unit tests to occur with system tests so that environment is builtgit status * generalizes natnet parameters and disables natnet automatically for launch * natnet client adaptor now references correct error code from NatNet SDK 4.4.0.0 * increment version tag * bug fixes to natnet launching from env file * fixed failing systems test due to depends issue and specifying unit tests via yaml * Use NatNet callback context instead of thread-local dispatch * addressing Krrish' documentation comments * incrementing version tag after osmo PR merge * documentation corrections * Bump VERSION Update .env --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Andrew Jong <ajong@andrew.cmu.edu>
* re-ordered initialization of stereo render product node to only initialize after right camera is initialized, ensuring camera is initialized as stereo (left camera is assumed, but right is optional) --------- Co-authored-by: John <johnliuchs2022@gmail.com>
* Add fixed-trajectory evaluation tests New tests/test_fixed_trajectory.py evaluates drone performance on Circle, Figure8, Racetrack, and Line trajectories: takeoff -> execute -> land with cross-track error, path RMSE, execution time, and success metrics recorded to metrics.json for baseline comparison. - Python ideal-path generators mirror fixed_trajectory_task.cpp equations - Cross-track error uses robot pose snapshot at dispatch to transform base_link ideal path to world frame for odom comparison - 5m loose tolerance documents the known circle failure without stranding drone - conftest.py gains --trajectory-types CLI option and generalised phase-order sorting/ID-rewriting for both autonomy test modules - tests/README.md documents the new module, all 11 metrics, and run commands Made-with: Cursor * Remove module docstring from test_fixed_trajectory.py Made-with: Cursor * Aj/GitHub ci cd (#347) * Add link to PAT * Change to new orchestrator instance workflow * Add availability zone * Bump version to 0.18.0-alpha.7 * Add fix for boot volume size blocking orchestrator * Add floating IPs to CI/CD * Bump gh runner_version to latest * Update cicd defaults * Rename integration-tests.yml to system-tests.yml * Add debugging tips and add to mkdocs * Use venv instead of pip3 to fix error: externally-managed-environment * Explicitly fail autonomy test if images not yet built * Enable using docker cache from docker registry to speed up docker image build tests for ci/cd * Fix bug * Update docs and change docker image build/push to also run on self-hosted runner * Enable trigger docker build workflow on via manual dispatch * Increase instance volume size so that space doesn't run out when building docker images * Update to always try build all images * Create dummy file for docker compose push to pass * Add omni_pass.env with guest access to AirLab nucleus * Update ci/cd tests to make sure image is present before running tests * Make sure images for profiles get built * Update system tests to not build images if pull available * Make build/pull quiet * Pin empy version to fix ROS2 jazzy version bug * Switch image to desktop so that tests run successfully * Add docker image signing to workflow * Change pytest mark 'autonomy' to 'takeoff_hover_land' * update comments on workflow * Recurisve checkout of airstack * Log more to GitHub * Better error logging for ci/cd orchestrator * Add check system resources before spawning server; if resources not available, report back and try again later * Make it so that pytest no longer triggers from pushes on PR; make it so we can manually trigger pytest by commenting /pytest * Update PR template * Update AGENTS.md * Fix finding baseline metrics * Update workflow to comment instead of react * Fix bug * Try fix another bug * Update omni_pass_TEMPLATE.env to use 'guest'; update default on system tests to include build_packages * Auto prepend 'build_packages' mark to ensure code is built before tests * Lower default stress-iterations to 1 and single takeoff-velocity to 0.5 * Johnliu/px4 cpu optimization (#348) * added option for physics step frequency * reverted example launch script * patches PX4 simulation startup script and fixes robot DDS version * set default physics Hz for PX4 to be 100Hz which is the minimum. * reverted simulation changes * updated docs * Better error logging for ci/cd orchestrator * Add check system resources before spawning server; if resources not available, report back and try again later * added option for physics step frequency * added option for physics step frequency * removed physics frequency from .env and set working PX4 values in docker-compose defaults. * removed unnecessary benchmarking from AirStack launch scripts. --------- Co-authored-by: Andrew Jong <ajong@andrew.cmu.edu> * Add new skills * Revise pull request template for clarity and detail Update pull request template with versioning guidelines Added guidelines for versioning in the pull request template. Update pull request template for media uploads Clarified instructions for adding videos and images in the PR template. * Johnliu/rtx lidar update (#351) * Update PegasusSim lidar to new rtx lidar and optional min_sensor_range parameter to vdb model to avoid self-detection. * removed deprecated ouster lidar. Completely integrated new rtx lidar * renaming frame id back to ouster * Added node to filter near and invalid lidar points * reconciled topic names for lidar point cloud * fixed example scripts to use rtx lidar api * fixed tmux closing and rclpy path issue * uses add_rtx in multi px4 script * bumping version index * docs added * unit testing and documentation updates * cleaning code from copilot suggestions * docs(tests): fix pytest marker example for running liveliness and sensors Agent-Logs-Url: https://github.com/castacks/AirStack/sessions/7ce7609a-a7f3-414d-9d42-0c9999d0459f Co-authored-by: andrewjong <8121216+andrewjong@users.noreply.github.com> * docs(tests): fix marker semantics in test_sensors module docstring Agent-Logs-Url: https://github.com/castacks/AirStack/sessions/bdf00f6f-1d9f-4597-bf57-b96f99421646 Co-authored-by: andrewjong <8121216+andrewjong@users.noreply.github.com> * addressing github copilot concerns * docs(bridge): remove stale camera topics comment Agent-Logs-Url: https://github.com/castacks/AirStack/sessions/2d5718ac-20e3-4f10-a12e-05d601cf000c Co-authored-by: JohnYanxinLiu <63010779+JohnYanxinLiu@users.noreply.github.com> * addressing copilot concerns * removing debug print statement from reading point cloud Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(isaac-sim): align drone1 lidar prim path with spawned prim Agent-Logs-Url: https://github.com/castacks/AirStack/sessions/fbad2b9c-1761-45b1-b464-3e874511255c Co-authored-by: JohnYanxinLiu <63010779+JohnYanxinLiu@users.noreply.github.com> * more succint comment in sim bashrc * resolving discrepant comments in ros bridge yaml * removed bug allocated new copy of point cloud array * logs lidaar test with boolean instead of hz * and --> or for marks --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: andrewjong <8121216+andrewjong@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Andrew Jong <ajong@andrew.cmu.edu> * Krrish/coord pr (#350) * Fixed multi-drone global plan * added sep files for fire and retro * added robot2 relative pos; diff rviz files; bridge for rayfronts topics * added sharing of semantic rays * changed rviz for both drones * added target sharing * changed drone start pos * gossip layer w/o relay * added global coords under /{ROBOT_NAME}/interface/mavros/global_position/raw/fix(not my topic, it was already publishing to that) * gossip, threedrone,peerprofile * multi drone vis in foxglove, odom doesn't work in foxglove yet * multi drone vis in foxglove works with odom * global plan added * added image, vdb markers(not transformed yet) * fixed state estimation flickering and vdb transform * added custom foxglove buttons for commands * added modular payloads to peerprofile, foxglove reads the payloads and vizualizes it,currently works for rayfronts * fixing the rotation of payload * syncing devices * fixed gossip + translate * added skill for foxglove/coordination * removed VDB ENV * rebase with main * updated docs * fixed launch files so they have play start on sim. scene_prep utils: added non-world prims to save in flattened manner * created raven_nav package * moved coordination to common * fixed gcs<->robot dds * added hitl functionality * fixes to dds * put dds hitl under gcs * fixes to robot hitl * syncing both computers * mimiced robot-l4t for dataflow * fixed path to ddsrouter_yaml * fixed dds server * fixed two_drone_fire * rayfronts is now a ros package * added feedback, it's sending success too early though * fixed raven behavior * foxglove panel with working executors * random walk fixed * fixed random walk bringup. Added saves and viz for multiple waypoints and polygons * fixed bounds for exploration task, combined waypoint/polygon editor into task panel * made waypoint/polygon gui larger * added 2d map to foxglove * WIP: pre-merge snapshot * added changes from main * WIP: pre-branch-split snapshot * PR for foxglove+multi-robot * merged with main * PR cleanup: revert unrelated changes and drop extra files - Restore main's robot.rviz (drop redundant robot_1/robot_2.rviz) - Restore ms-airsim include in root docker-compose.yaml - Restore airsim sections in docs/simulation/index.md - Restore docs/gcs/docker/index.md (VERSION env name) - Restore robot/docker/{.bashrc, Dockerfile.robot} to main - Restore SIM_IP in robot/docker/docker-compose.yaml - Restore takeoff_landing_planner takeoff_height: 8.0 - Drop docs/action_bridging.md (internal design memo) - Drop personal launch scripts (two_drone_fire*, three_drone_scene_import, two_drone_RetroNeighbourhood) - Trim verbose comments in gps_utils.py and example_multi_drone_scene_import.py * Trim noisy inline comments in PR-added Python files * Pin vdb_mapping_ros2 to public main (was at unpushed 68fe8dde) * fixed launch script * fixed foxglove bugs, added dynamic fg layout, updated docs * fixed bugs found by copilot. Removed rviz by adding a node * fixed comment Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fixed path in skill Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fixes from copilot * Fix Pegasus submodule pointer after merge Advance to 8e01d013 (main's pointer) which contains spawn_rtx_lidar.py, required by example_one_px4_pegasus_launch_script.py and the multi script after the rtx-lidar update merged from main. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(coordination): align gossip with steady clock + manifest hygiene - gossip_node: swap startup log + outgoing-stamp clock to STEADY_TIME so the dedup-by-stamp invariant survives /clock pauses; subscribe to /global_position/global to match foxglove_visualizer and action_relay - gossip_node docstring: drop the false "waypoint triggers immediate publish" claim - coordination README: rename peer_registry node block to the actual per-robot registry topic; "wall-clock" -> "steady" - package.xml: add missing exec/depend rules - coordination_bringup -> autonomy_bringup - autonomy_bringup -> coordination_bringup - desktop_bringup -> coordination_bringup, gcs_visualizer - gcs_visualizer -> std_msgs, coordination_msgs, coordination_bringup - task_msgs: replace TODO license with BSD-3-Clause - gcs.launch.xml: comment had `--no-sandbox` (`--` is illegal inside an XML comment and crashed the ROS launch parser) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(gcs+autonomy): drop dead BT panel, lint payload imports, name-map override - payload_visualizer_node: remove unused PointCloud2 / transform_point_cloud2 imports (F401), collapse Marker/MarkerArray - action_relay launch: ROBOT_RELAY_MAP env override for non-default robot_name -> domain mappings (default behavior unchanged) - desktop_bringup robot.rviz: drop BehaviorTreePanel entry pointing at /behavior/behavior_tree_graphviz (publisher package was removed) - autonomy_bringup domain_bridge: bridge /global_position/global to match the dds_router and the rest of the stack (was /raw/fix) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(foxglove): clean panel-id stacking, atomic render, drop dead .foxe - render_layout: regex now strips every trailing _r<n> (was: only the last one), fixes _r1_r1_r1... stacking on repeated runs - render_layout: atomic write via tmp + os.replace so a partial json.dump doesn't corrupt the layout file - airstack_default.json: re-render with fixed stripper to commit a clean source template (no stacked _r1 suffixes) - install.sh -> install.py: file is Python, shebang is python3 - install.py: slugify publisher into the on-disk extension dir name so "AirLab CMU" doesn't produce a directory with a space - drop robot-commands/robot-commands.foxe (duplicate; canonical is at foxglove_extensions/robot-commands.foxe) and the .foxe.bak Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * bug fixes * bug fixes * version * reverted env * updated gitignore and docs * updated foxglove viz + consistent spellings across repo * Move layout file to /root/ so it's immediately accessible, also fix template path * Change so that file name reflects NUM_ROBOTS * Add a DEBUG_RVIZ flag to launch robot rviz if needed --------- Co-authored-by: krrishj18 <krrishj18@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Andrew Jong <ajong@andrew.cmu.edu> * Scene prep bug fix (#354) * fixes to scene_prep_utils.py * edited docs * clean launch script * updated version * fixed comments inconsistency and typos * formatting fix Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * bug in gossip if payload is empty * fixed omni_pass.env file creation bug from CICD guest default profile * fixed depth topic naming in foxglove gcs * changed gps topic * removed redundant exntentions --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: airlab <johnliuchs2022@gmail.com> * Add workflows to (1) enforce correct branch merge convention (2) update develop from main * Update docs on branches * Update workflow to handle develop version increment * Release 0.18.0 * Bump VERSION to after sync from main * feat(osmo): VS Code/Cursor dev workflow on NVIDIA OSMO (#352) * feat(osmo): VS Code/Cursor dev workflow on NVIDIA OSMO Adds a privileged Docker-in-Docker workspace task that lets a developer run the full AirStack docker-compose stack on OSMO and attach an IDE over SSH, with Isaac Sim WebRTC livestream + Foxglove websocket exposed via osmo port-forward. Components: - osmo/workspace/{Dockerfile,entrypoint.sh,sshd_config}: airstack-osmo-workspace image. Ubuntu 24.04 + sshd (pubkey-only) + Docker CE + Docker Compose + nvidia-container-toolkit + fuse-overlayfs (DinD-on-overlayfs needs it, otherwise dockerd falls back to vfs which bloats AirStack images ~10x). - osmo/workflows/airstack-dev.yaml: single privileged GPU task. Materializes Nucleus + airlab-docker secrets from OSMO credentials, clones AirStack, starts inner dockerd, runs `airstack up` with desktop + isaac-sim-livestream Compose profiles. - simulation/isaac-sim: isaac-sim-livestream Compose service that runs Pegasus standalone with --/app/livestream/enabled=true and exposes WebRTC port ranges 47995-48012 / 49000-49007 / 49100; launch script gates headless+livestream extension on ISAAC_SIM_LIVESTREAM env var. - .airstack/modules/osmo.sh: airstack osmo:{up,ide,foxglove,webrtc,logs,down} CLI wrappers around `osmo workflow submit` / `port-forward` / `cancel`. Persists the active workflow id and validates it's still running before each command (prevents the stale-state 410 error). - airstack.sh: bash 4+ re-exec bootstrap (macOS ships 3.2; the CLI uses `declare -A`). - osmo/README.md + docs/tutorials/airstack_on_osmo.md: admin pool setup (privileged_allowed) + per-user credentials (airlab-docker-login, airlab-nucleus) + student-facing IDE attach + WebRTC/Foxglove flow. Pool requirements: privileged_allowed: true, GPU pool with nvidia-container-toolkit on the host, ample node ephemeral storage (AirStack images extracted are ~50-100Gi via fuse-overlayfs; vfs needs ~500Gi+). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(osmo): harden CLI + workspace image against stale-state, port-forward race, and cursor-server install hangs Four bugs that bit the first end-to-end runs (airstack-dev-10 → -13): - _osmo_wf_id: validate saved workflow id against `osmo workflow query` before returning. Without this, the state file at ~/.airstack/osmo-state outlives the workflow it points at and every subsequent osmo:webrtc / osmo:foxglove / osmo:ide call surfaces the same confusing "Workflow airstack-dev-N is not running! (status 410)" instead of the obvious "run airstack osmo:up to launch a fresh workflow". - cmd_osmo_up: `osmo workflow submit --set-env` is variadic. Passing two separate `--set-env A=1 --set-env B=2` silently drops the first one — this is what made airstack-dev-11 fail with "ERROR: SSH_PUB_KEY not set" when --branch was passed alongside the pubkey. Collapse the K=V pairs into a single --set-env. - cmd_osmo_ide: previously launched the IDE before starting the port-forward, so Cursor/VS Code would try to SSH localhost:2200 a few hundred ms before the tunnel listener existed and fail with "connect to host localhost port 2200: Connection refused". Now: detect an existing forward and reuse it (also avoids the "Address already in use" if osmo:foxglove was started in parallel), otherwise spawn the forward in the background, wait up to 30s for it to bind, then launch the IDE. Ctrl+C tears down the spawned forward cleanly via a trap. - workspace image / entrypoint: Cursor Remote-SSH hung indefinitely on airstack-dev-13 because (a) cursor-server's installer fell back to wget when curl timed out and wget was not in the image, and (b) a /tmp/cursor-remote-lock.* file left behind by the first crashed install blocked every silent retry. Add wget to the apt install list and rm -f the stale Cursor / VS Code remote lock files at the very top of entrypoint.sh so each fresh pod starts from a clean slate. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(osmo): correct osmo:logs CLI invocation; install Foxglove extensions locally on osmo:foxglove osmo:logs was invoking `osmo workflow logs <id> workspace --follow`, but the real CLI takes the task via `-t TASK` (not positionally) and has no `--follow` flag at all — so the command failed immediately with "unrecognized arguments: workspace --follow". Replace with a polling loop that uses `-t workspace -n <N>` on a short interval, prints only the suffix that appeared since the previous fetch (find-the-last-seen-line trick; degrades to "reprint tail" with a warning if the cursor outruns -n), and exits cleanly once the workflow reaches a terminal state. Tunables: OSMO_LOGS_TASK / OSMO_LOGS_TAIL / OSMO_LOGS_INTERVAL. osmo:foxglove now installs the AirStack Foxglove extensions (robot-commands / waypoint-editor / polygon-editor) into the laptop's local Foxglove user-extensions directory before opening the port-forward. Without this, custom panels show up as "Unknown panel type: robot-commands.Robot Tasks" in the laptop's Foxglove Desktop because it has no way to discover the extension folders that live inside the GCS container. To avoid duplicating the install logic, the existing gcs/foxglove_extensions/install.py is refactored to read FOXGLOVE_EXT_SRC / FOXGLOVE_EXT_DST env vars (the in-container call already in gcs/docker/gcs-base-docker-compose.yaml keeps working unchanged via defaults). The wrapper sets those vars to ${PROJECT_ROOT}/gcs/foxglove_extensions and ~/.foxglove-studio/extensions respectively, overridable with OSMO_FOXGLOVE_EXT_DIR / skippable with OSMO_FOXGLOVE_SKIP_EXTENSIONS=1. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(osmo): pin Kit livestream UDP media port to 49099 so osmo:webrtc actually shows pixels Kit 107's WebRTC livestream picks a UDP media port dynamically. The documented `omni.services.livestream.nvcf` defaults (minHostPort=47998 maxHostPort=48020 fixedHostPort=0) are ignored by the stock standalone Kit binary — on airstack-dev-13 it bound to UDP 49042, outside both the Compose-published range AND the default `osmo:webrtc --udp` forward of `47995-48012,49000-49007`. Result: TCP signaling on 49100 worked, the WebRTC Streaming Client window opened, but every SRTP media packet was dropped → black viewport plus the recurring `NVST_CCE_DISCONNECTED when m_connectionCount 0 != 1` underflow in Kit's log. Pin the media port via three `app.livestream.*` settings set on `SimulationApp` before `omni.kit.livestream.webrtc` is enabled, so whichever code path the carb.livestream-rtc.plugin consults lands on the same port: app.livestream.fixedHostPort = 49099 app.livestream.minHostPort = 49099 app.livestream.maxHostPort = 49099 49099 is a deliberate one-off from the 49100 TCP signaling port — same neighborhood, easy to remember. Verified live on airstack-dev-13 after `docker compose up -d --force-recreate isaac-sim-livestream`: Kit binds UDP 49099 (`/proc/net/udp` hex BFCB on 0.0.0.0) and docker-proxy publishes it from the pod host network. Knock-on cleanups: - `simulation/isaac-sim/docker/docker-compose.yaml` shrinks the isaac-sim-livestream `ports:` from 27 forwarded ports (`47995-48012, 49000-49007 TCP+UDP, 49100 TCP`) to just two: `49100/tcp` + `49099/udp`. - `.airstack/modules/osmo.sh` shrinks `OSMO_WEBRTC_TCP` to `49100` and `OSMO_WEBRTC_UDP` to `49099`, so `airstack osmo:webrtc` spawns two port-forwards instead of thirty. - `.gitignore` ignores `.DS_Store` so working from a Mac doesn't leak Finder metadata. After pulling this commit into a running pod: `docker compose up -d --force-recreate isaac-sim-livestream` to apply the new port mapping; then re-run `airstack osmo:webrtc` on the laptop to pick up the new forward ranges. The standalone WebRTC Streaming Client connects to `localhost` (same address as before) and now actually receives frames. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(osmo): render Kit GUI in WebRTC stream; document SSH agent forward for in-pod git push Two paper-cuts that bit airstack-dev-13 after the WebRTC media port pin landed (commit 2d9b161): (1) The WebRTC stream showed only the bare 3D viewport — no menu bar, no toolbar, no panels, no console. Cause: SimulationApp's default when `headless=True` is to also hide the UI (`hide_ui=True`). The NVIDIA reference at `simulation/isaac-sim/standalone_examples/api/isaacsim.simulation_app/livestream.py` explicitly opts back into UI rendering plus picks explicit window sizing and `display_options=3286` to keep the default grid/axes visible. Mirror that config in `example_one_px4_pegasus_launch_script.py` when `ISAAC_SIM_LIVESTREAM=true` (local desktop dev keeps the minimal `headless=False` path unchanged). (2) The pod has no SSH private key, only an `authorized_keys` for inbound connections from the user's laptop. As a result, `git push` from inside the Cursor / VS Code Remote-SSH session inside the pod fails with "Permission denied (publickey)". sshd inside the workspace image already has `AllowAgentForwarding yes` baked in via `osmo/workspace/sshd_config`; the missing piece is purely on the Mac side. Update the `~/.ssh/config` block in the tutorial to include `ForwardAgent yes` (so the local agent's keys are exposed in the pod), `AddKeysToAgent yes` (auto-load on first push), and `UseKeychain yes` (macOS-only Keychain unlock without passphrase prompts; ignored on Linux). Adds an `ssh-add -l` smoke-test note. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(osmo): make osmo:setup idempotent + paste-safe; document Nucleus auth-debug path osmo:setup hit two failure modes that wasted a debug session each: - `osmo credential set` is not an upsert for GENERIC creds — re-running setup (e.g. to rotate a Nucleus API token) failed with `400 duplicate key value violates unique constraint "credential_pkey"` and bailed before reaching the airlab-nucleus credential. Delete-then-set each credential so re-running is idempotent. - Bracket-paste mode and cross-OS clipboards routinely smuggle invisible bytes around long pastes. Nucleus's auth endpoint silently DENIES a token with one extra trailing byte, with no actionable error from the client side. _osmo_prompt now strips leading/trailing whitespace and CR/NUL bytes via a new _osmo_trim helper, and warns when bytes were stripped. cmd_osmo_setup additionally JWT-shape-checks the Nucleus token (must be eyJ.<dot>.<dot>.) before submitting it, so a wrong paste fails at setup time instead of silently DENIED at pod boot. Also documents how to debug the "Login Required: Unable to connect server omniverse://airlab-nucleus..." popup: SSH the Nucleus host and tail base_stack-nucleus-auth-1 for InternalCredentials.auth status: DENIED. Adds a "Nucleus connectivity from OSMO" section to the admin README clarifying that Nucleus over HTTPS uses a single 443 (no need to open the native 3009-3180 range from the OSMO cluster), per NVIDIA's TLS docs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(osmo): use Nucleus API-token auth, with double-dollar to survive compose parser The OSMO entrypoint was writing OMNI_USER=<andrew_id> alongside an API token JWT in OMNI_PASS, which routes the JWT through the password- verification path. Nucleus silently DENIES — visible only in base_stack-nucleus-auth-1 as `InternalCredentials.auth … 'username': '<andrew>' … status: DENIED` (no Tokens.auth_with_api_token call). Kit then pops "Login Required: Unable to connect server omniverse://...". omniclient expects the literal sentinel username `$omni-api-token` paired with the JWT as the password. The entrypoint now detects a JWT-shaped OMNI_PASS (header starts with `eyJ`) and emits OMNI_USER=$$omni-api-token into omni_pass.env. The `$$` is intentional: docker-compose v2 interpolates env_file values, and a single `$` would be eaten by the parser (`OMNI_USER=$omni-api-token` becomes `OMNI_USER=-api-token` after ${omni}- expansion to empty). The container ultimately sees OMNI_USER=$omni-api-token, which is the correct sentinel. Also note for the next debugger: `docker compose restart` does NOT re-read env_file. Use `docker compose up -d <svc>` to recreate the container after editing omni_pass.env. Updates omni_pass_TEMPLATE.env header to document the API-token pattern explicitly (with the $$ caveat), and adds a troubleshooting row that distinguishes "wrong auth path" (DENIED with no Tokens.auth_with_api_token call) from "bad/expired token" (Tokens.auth_with_api_token: DENIED). Co-authored-by: Cursor <cursoragent@cursor.com> * docs(osmo): make OSMO the recommended dev path, single clone-the-repo flow Reposition the OSMO tutorial as AirStack's recommended day-to-day development path (not just a fallback for laptops without GPUs) and collapse it onto a single recipe: clone the repo, then drive everything through the airstack osmo:* wrappers in .airstack/modules/osmo.sh. - docs/tutorials/airstack_on_osmo.md - Retitle + rewrite the intro to lead with five concrete advantages (pooled GPUs, no local CUDA/Docker/driver maintenance, same image as CI + field robots, one-command onboarding, hardware bigger than your laptop). Demote the Linux+GPU-desktop path to an escape hatch. - Drop the Mac/Windows/no-GPU framing in 'Who is this for?' and the mermaid laptop subgraph label. - Add 'a local clone of AirStack' to Prerequisites; remove it from the 'do not need' list. - Replace Option A/B credential split with a single ./airstack.sh osmo:setup recipe; move the three raw osmo credential set calls into a collapsible 'Under the hood' footnote. - Replace each step's raw osmo workflow ... command with the corresponding airstack osmo:up/logs/ide/webrtc/foxglove/down wrapper; preserve the raw form in 'Under the hood' footnotes that cross-link cmd_osmo_* in .airstack/modules/osmo.sh. - Drop the export WF=... paragraph — the wrappers read the id from ~/.airstack/osmo-state automatically; AIRSTACK_OSMO_WF overrides per-invocation. \$WF now only appears inside the raw-form footnotes. - Sweep Troubleshooting + What-survives tables: redirect raw port-forward fixes to the airstack osmo:* equivalents and rename the section to 'What survives airstack osmo:down?'. - Fix WebRTC edge label (49100/tcp + 49099/udp) to match the pinned ports the workflow actually uses today. Companion cleanups now that the privileged_allowed flip is automatic on the OSMO autosync side (synchronize_osmo_team_pools.py forces privileged_allowed: true on every platform of every pool, so students never see the 'platform does not have privileged flag enabled' error): - osmo/README.md: drop the 'Most common blocker' privileged warning, the privileged_allowed row from the pool-requirements table, and the 'privileged GPU pod' / '(privileged, GPU)' descriptors in the architecture summary. Simplify the validation-stage SSH-failure hint. - osmo/workflows/airstack-dev.yaml: trim the long DinD-requires-privileged comment to a one-liner (the privileged: true directive itself stays). - .airstack/modules/osmo.sh: remove the special-case 'privileged flag enabled' error branch in cmd_osmo_up — it should never fire now. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(osmo): make osmo:logs actually stream + survive pod host-key churn osmo:logs was silent because cmd_osmo_logs wrapped osmo workflow logs in $( ... ) on the assumption that -n LAST_N_LINES exits after dumping the tail. Empirically the CLI keeps the stream open as new lines arrive (it already behaves like tail -f, despite --help advertising only -n), so command substitution waited forever and printed nothing. Drop the polling loop and just exec the command directly. Each fresh OSMO pod also ships a new sshd host key, so every osmo:up trips StrictHostKeyChecking against the previous workflow's fingerprint and SSH/Cursor abort with "Host key for [localhost]:2200 has changed". Switch the recommended ~/.ssh/config block (and osmo/README.md) to the ephemeral-host pattern (StrictHostKeyChecking no + UserKnownHostsFile /dev/null + LogLevel ERROR), and have cmd_osmo_ide ssh-keygen -R the stale loopback entry on every run so users on the old config get unblocked automatically. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(osmo): auto-pin --branch to local checkout + clean error UX when workflow dies The pod's entrypoint clones AirStack fresh from GitHub on every workflow start (the pod fs is ephemeral). It defaulted to `main`, so any developer testing branch-only OSMO changes silently ran their pod against stale `main` code — most visibly: COMPOSE_PROFILES=desktop,isaac-sim-livestream resolved to "desktop" alone on `main` because the isaac-sim-livestream service only exists on the feature branch, so isaac-sim never came up and `airstack osmo:webrtc` showed a blank stream. - cmd_osmo_up now defaults --branch to the local repo's current branch (git rev-parse --abbrev-ref HEAD). Detached HEAD or non-git checkouts fall back to `main` cleanly. Pass --branch explicitly to override. - New _osmo_check_branch_pushed warns up-front when the about-to- submit branch has no upstream, is ahead of origin, or has an uncommitted working tree. The pod doesn't see your laptop's edits. Separately, when an OSMO workflow gets canceled mid-flight (osmo:down in another shell, or OSMO timing it out), the in-flight port-forward and logs streams raise OSMOUserError("Workflow X is not running!") from inside an asyncio Task. The CLI prints "Task exception was never retrieved" + a multi-line Traceback that buries the actual one-line cause. New _osmo_pf_filter awk script collapses that into a single [ERROR] line pointing at `airstack osmo:up`. Wired into webrtc, foxglove, and logs. webrtc also gains a cleanup trap that kills the backgrounded UDP port-forward on EXIT/INT/TERM so we don't leak it against a dead workflow. Tutorial Step 2 documents the new --branch default and the "pod-clones-from-GitHub-not-your-laptop" gotcha. Co-authored-by: Cursor <cursoragent@cursor.com> * perf(osmo): bump inner dockerd concurrency to saturate 10 GbE pulls dockerd's defaults of --max-concurrent-downloads=3 / --max-concurrent -uploads=5 cap a fresh airstack-dev pod's image-pull at ~300 MiB/s against the airlab-backup-10g registry — single-stream TLS tops out around 300-500 MiB/s per core, and three parallel streams of unevenly sized blobs serialize down to that ceiling. Ceph (1014 TiB, 92 OSDs, SSD pools) and 10 GbE both have far more headroom than that. Bump to 10/10 to overlap enough blob downloads to saturate the pipe. Threaded through the DOCKERD_MAX_DOWNLOADS / DOCKERD_MAX_UPLOADS env vars so a pool can be tuned at submit time without rebuilding the workspace image. Workspace image needs a rebuild + push for this to take effect: cd osmo/workspace docker build -t airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest . docker push airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest Co-authored-by: Cursor <cursoragent@cursor.com> * docs(osmo): require buildx --platform linux/amd64 for workspace image A plain `docker build && docker push` on an Apple Silicon Mac silently produces a linux/arm64-only `latest` manifest. OSMO workers are amd64, so every subsequent workflow fails at the outer pod-image pull with "no match for platform in manifest" before the entrypoint even runs — a confusing failure mode whose root cause lives entirely in the push, not in the workflow yaml or the entrypoint. Switch the README and the Dockerfile docstring to the buildx form, explain the why, and document the post-push manifest check. Co-authored-by: Cursor <cursoragent@cursor.com> * perf(osmo): move dockerd data-root to /osmo/run for native overlay2 The OSMO pod's `/` is itself a containerd overlay snapshot, and Linux refuses to stack a second overlayfs on top of an overlay rootfs — which is why the inner dockerd was falling through to fuse-overlayfs. That costs a kernel↔userspace FUSE round-trip on every `creat()` during layer extraction, which murders throughput on apt/pip/ROS layers (measured: 32-50 MB/s for small-file-heavy layers vs 480 MB/s for big-file layers in the same pull). Pointing dockerd at /osmo/run/docker (the kubelet emptyDir backed by ext4 on /dev/vda3) lets the existing overlay2-first fallback chain actually succeed on its first try, restoring kernel-overlay extraction performance. emptyDir lifetime matches the workflow lifetime, so the docker layer cache gets the right scope automatically. Falls back to /var/lib/docker if /osmo/run isn't present so the image still works in non-OSMO test contexts. Co-authored-by: Cursor <cursoragent@cursor.com> * updated version * added virtual display for GL context * added virtual display for droan_gl * droan_gl patch * run Xvfb in its own tmux session * updated dockerfile + version * typo in docs Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in comments Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in comments Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in comments Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in osmo logs, renamed airstack-isaac-sim to just isaac-sim Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in container name for isaac-sim-livestream Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * airstack-dev version overwrite removed Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: krrishj18 <krrishj@andrew.cmu.edu> Co-authored-by: Andrew Jong <ajong@andrew.cmu.edu> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add fixed-trajectory evaluation tests New tests/test_fixed_trajectory.py evaluates drone performance on Circle, Figure8, Racetrack, and Line trajectories: takeoff -> execute -> land with cross-track error, path RMSE, execution time, and success metrics recorded to metrics.json for baseline comparison. - Python ideal-path generators mirror fixed_trajectory_task.cpp equations - Cross-track error uses robot pose snapshot at dispatch to transform base_link ideal path to world frame for odom comparison - 5m loose tolerance documents the known circle failure without stranding drone - conftest.py gains --trajectory-types CLI option and generalised phase-order sorting/ID-rewriting for both autonomy test modules - tests/README.md documents the new module, all 11 metrics, and run commands Made-with: Cursor * Spherical lookahead bug that fixed the circle test and caused the circle test to pass * Added in code that consolidated all the results code so the user can easily see their results in one file without having to wade through a ton of log files to get what they need * Results for 10 tries headless summary statistics * Fixed the logging files so now it only outputs one summary file and it doesn't inundate the user with a ton of log files for no reason * deleted cleanup_old_results.sh which was a local tool for cleaning up everything * Added preliminary docs to explain changes made * Changed .env to say 0.19.0-alpha.4 * Resolved all the merge conflicts that are in this file * Revert sphere_radius to 1.0; velocity_sphere_radius_multiplier=1.0 makes the fixed value inert Co-authored-by: Cursor <cursoragent@cursor.com> * Remove internal branch reference from baseline; note AirStation hardware Co-authored-by: Cursor <cursoragent@cursor.com> * Remove parameter tuning bullet from docs after reverting sphere_radius Co-authored-by: Cursor <cursoragent@cursor.com> * Move system-test prerequisites to index.md and reference it from fixed-trajectory doc Co-authored-by: Cursor <cursoragent@cursor.com> * Remove path tracker bug fixes section from docs (covered in PR description) Co-authored-by: Cursor <cursoragent@cursor.com> * Trim duplicated stack bring-up from manual usage; link to Getting Started Co-authored-by: Cursor <cursoragent@cursor.com> * Reframe fixed-trajectory doc as end-to-end testing guide Rename fixed_trajectory_testing.md to end_to_end_testing.md (history preserved), add e2e intro and future-work note, fix stale test path to tests/system, and update mkdocs nav, testing index, and tests/README references. Co-authored-by: Cursor <cursoragent@cursor.com> * removed stale test_sensors file * incremented version tag * Fixed the summary.txt file after it broke after a ton of commits were completed. * resyncing Pegasus module to fixed camera initialization fix --------- Co-authored-by: pvkumara <pkumara@andrew.cmu.edu> Co-authored-by: Andrew Jong <ajong@andrew.cmu.edu> Co-authored-by: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: andrewjong <8121216+andrewjong@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Krrish Jain <krrishj@andrew.cmu.edu> Co-authored-by: krrishj18 <krrishj18@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: airlab <johnliuchs2022@gmail.com> Co-authored-by: Andrew Jong <andrewjong@fieldai.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Sebastian Scherer <basti@andrew.cmu.edu> Co-authored-by: Cursor <cursoragent@cursor.com>
…ution fixes (#370) Foundational real-robot deployment fixes extracted from the OptiTrack emulation PR (#367) so they can be reviewed and merged first; #367 will be rebased on top afterward, shrinking its diff. Docker / ARM build: - Add TARGET_ARCH build arg (default x86_64) to Dockerfile.robot and use it to parametrize LD_LIBRARY_PATH, so the aarch64 (Jetson/l4t, voxl) images link against the correct arch triplet. - docker-compose.yaml passes TARGET_ARCH: aarch64 to the voxl and l4t image builds. - Install ros-${ROS_DISTRO}-mavros-extras (generic dep; also provides the vision_pose plugin used by external-pose deployments). Robot name resolution: - .bashrc now follows a pre-set ROBOT_NAME (e.g. injected by docker compose) instead of always overriding it from the container/hostname mapping. The bws() flock build lock is retained. - default_robot_name_map.yaml catch-all fallback maps to unknown_robot (valid ROS namespace token) instead of unknown-robot. Version bumped 0.19.0-alpha.5 -> 0.19.0-alpha.6 for the version-increment gate. Note: the trajectory_controller/trajectory_library robustness fixes originally listed for extraction are already present on develop (PR #365), so they are not included here. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…rdware (#371) * feat(l4t): make robot-l4t deployment knobs overridable + document name resolution Parametrize the robot-l4t compose service so a single service covers real deployments without editing compose: - AUTONOMY_ROLE and FCU_URL are now ${VAR:-default} overridable (and FCU_URL is unquoted so the literal serial path reaches mavros). - Rosbag output path is BAG_STORAGE_PATH-overridable. Update the configure-multi-robot skill to reflect the honor-pre-set-ROBOT_NAME guard (#370): document pinning ROBOT_NAME in an override for a single real robot, the never-on-the-shared-service caveat, and the unknown_robot fallback fixes by topology. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(l4t): add site-agnostic l4t-px4-realrobot override template Deployment override for a single real PX4 robot on a Jetson (aarch64/l4t). Surfaces the common knobs at the top with sensible defaults: ROBOT_NAME pinned directly (single-robot shortcut honored by .bashrc), FCU_URL, AUTONOMY_ROLE, BAG_STORAGE_PATH, and RECORD_BAGS. Mocap-agnostic — NatNet/external-vision settings are added by a separate optitrack override. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(l4t): entrypoint passthrough + ZED SDK 5.2; document build gotchas Two real-hardware build fixes for the Jetson profile: - Dockerfile.l4t-stack-base: overwrite dustynv's /ros_entrypoint.sh with an `exec "$@"` passthrough. Its prebuilt source-ROS libs (fastcdr 2.2.5) were shadowing the apt Jazzy (2.2.7) that Dockerfile.robot layers on, crashing apt-built nodes like mavros with symbol-lookup errors under tmux autolaunch. - zed/Dockerfile.zed-l4t: bump ZED SDK 4.2 -> 5.2 and move the coupled ROS deps together (zed_msgs 5.2.1, point_cloud_transport(_plugins) 4.x, add backward_ros). Document both gotchas in the docker-build-profiles skill, and correct the stale unknown-robot -> unknown_robot in the robot_identity reference doc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: bump version to 0.19.0-alpha.7 Version-increment gate: bump above develop's 0.19.0-alpha.6 and record the l4t deployment changes in the changelog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#372) * feat(l4t): make robot-l4t deployment knobs overridable + document name resolution Parametrize the robot-l4t compose service so a single service covers real deployments without editing compose: - AUTONOMY_ROLE and FCU_URL are now ${VAR:-default} overridable (and FCU_URL is unquoted so the literal serial path reaches mavros). - Rosbag output path is BAG_STORAGE_PATH-overridable. Update the configure-multi-robot skill to reflect the honor-pre-set-ROBOT_NAME guard (#370): document pinning ROBOT_NAME in an override for a single real robot, the never-on-the-shared-service caveat, and the unknown_robot fallback fixes by topology. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(l4t): add site-agnostic l4t-px4-realrobot override template Deployment override for a single real PX4 robot on a Jetson (aarch64/l4t). Surfaces the common knobs at the top with sensible defaults: ROBOT_NAME pinned directly (single-robot shortcut honored by .bashrc), FCU_URL, AUTONOMY_ROLE, BAG_STORAGE_PATH, and RECORD_BAGS. Mocap-agnostic — NatNet/external-vision settings are added by a separate optitrack override. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(l4t): entrypoint passthrough + ZED SDK 5.2; document build gotchas Two real-hardware build fixes for the Jetson profile: - Dockerfile.l4t-stack-base: overwrite dustynv's /ros_entrypoint.sh with an `exec "$@"` passthrough. Its prebuilt source-ROS libs (fastcdr 2.2.5) were shadowing the apt Jazzy (2.2.7) that Dockerfile.robot layers on, crashing apt-built nodes like mavros with symbol-lookup errors under tmux autolaunch. - zed/Dockerfile.zed-l4t: bump ZED SDK 4.2 -> 5.2 and move the coupled ROS deps together (zed_msgs 5.2.1, point_cloud_transport(_plugins) 4.x, add backward_ros). Document both gotchas in the docker-build-profiles skill, and correct the stale unknown-robot -> unknown_robot in the robot_identity reference doc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: bump version to 0.19.0-alpha.7 Version-increment gate: bump above develop's 0.19.0-alpha.6 and record the l4t deployment changes in the changelog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(infra): collect co-located unit tests via the package list + integration tier Unit tests are defined by tests/colcon_unit_test_packages.yaml: conftest.py resolves each listed package to its <pkg>/test dir and collects the non-linter test_*.py files under --import-mode=importlib (set in pytest.ini), marking each `unit` by path. ament lint files are skipped (they run under colcon test). Removes two now-unnecessary files under tests/robot/; the package test/ dirs are collected directly. Also add an integration test tier: tests/integration/ + `integration` mark + a shared robot_autonomy_stack fixture (robot-desktop container, no sim/GPU), slotted into _MODULE_ORDER between build_packages and the sim tiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(testing): describe unit tests as co-located and listed in the package YAML Update the add-unit-tests and run-system-tests skills, AGENTS.md, and the unit-testing docs: adding a unit test is "list the package in colcon_unit_test_packages.yaml", and the source lives in the package's own test/ dir. Document the `integration` mark/tier. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: bump version to 0.19.0-alpha.8 Version-increment gate: bump above develop (0.19.0-alpha.6); alpha.7 is taken by the l4t-deployment-fix PR. Record the test-infra changes in the changelog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(tests): split unit-test discovery + session state into tests/harness/ Begin modularizing conftest.py (959 lines) by concern. Extract two self-contained pieces into a new tests/harness/ package: - harness/session.py: session-scoped mutable state (results dir, current pytest item, last subprocess output, logger) with setter/getter accessors. Hooks write it; helpers read it, so helper modules no longer reach into conftest globals. - harness/discovery.py: unit-test discovery driven by colcon_unit_test_packages.yaml (repo_path, load_colcon_unit_test_config, colcon_test_robot_command, unit_test_dirs, unit_test_files, _is_unit_item). conftest.py imports from harness and its hooks delegate to the session accessors; it re-exports AIRSTACK_ROOT / colcon_test_robot_command / load_colcon_unit_test_config / logger so existing `from conftest import ...` in the system tests keeps working unchanged. Behavior-preserving (host-validated): `-m unit` still 14 passed / 152 deselected, 166 collected, same order. Follow-on: the commands/containers/metrics/sim helpers and collection ordering move out the same way. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(tests): extract commands/containers/metrics/sim helpers into tests/harness/ Continue modularizing conftest.py. Move the subprocess/ros2 command helpers (harness/commands.py), docker container + compute-usage + image helpers (harness/containers.py), MetricsRecorder + get_metrics/current_test_id (harness/metrics.py), and the sim target configs + ros2 topic sampling (harness/sim.py) out of conftest.py. conftest.py drops from 836 to 360 lines and re-exports the harness helper API (`from harness import *`) so `from conftest import <name>` in the system tests + sensor_probes keeps working unchanged. Behavior-preserving: -m unit still 14 passed / 152 deselected, 166 collected, same order. Remaining in conftest: pytest hooks, collection ordering, and the airstack_env / robot_autonomy_stack fixtures. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(tests): extract collection ordering into tests/harness/collection.py Final step of the conftest.py modularization: move test ordering — _MODULE_ORDER, the per-module phase chains, _module_key, and the parametrize-id rewrite — into harness/collection.py. conftest's pytest_collection_modifyitems hook now delegates to collection.modify_items(items). conftest.py is now 246 lines (from 959): pytest hooks + the airstack_env / robot_autonomy_stack fixtures. All helpers live in tests/harness/ by concern (session, discovery, commands, containers, metrics, sim, collection). Behavior-preserving: -m unit still 14 passed / 152 deselected, 166 collected, unit → build → integration → sim order unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Also sync docs/skills to the tests/harness/ layout (AGENTS.md, tests/README.md, tests/integration/README.md, run-system-tests + add-unit-tests skills, unit_testing + end_to_end_testing docs): helpers, MetricsRecorder, the workspace globs, and _MODULE_ORDER now point at tests/harness/ instead of conftest.py (still re-exported via conftest). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(robot): pin pytest to 7.4.* so apt launch_pytest stays compatible The builder-stage pip block pulled pytest >=8 transitively into /usr/local (copied into the runtime image), shadowing Jazzy's apt python3-pytest 7.4. pytest 8 removed the `path` argument from pytest_pycollect_makemodule, which apt's launch_pytest plugin still declares — so every pytest invocation in the robot container aborted at plugin registration. This broke `colcon test` for ament_python packages (e.g. lidar_point_cloud_filter in test_colcon_test_robot), while ament_cmake gtest packages were unaffected. Pin pytest to Jazzy's version so the container is internally consistent and launch_testing / launch_pytest remain usable for future launch-based tests. The test runner (tests/docker) is a separate interpreter and keeps its newer pytest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(isaac-sim): clear LD_LIBRARY_PATH for PX4 ubuntu.sh so ca-certificates configures The global ENV LD_LIBRARY_PATH puts isaac-sim's bundled libs (.../isaacsim.ros2.bridge/jazzy/lib) on the linker path. Its older libcrypto.so.3 shadows the system one, so when the updated ca-certificates (20240203 → 20260601~24.04.1) runs its postinst `openssl`, it fails with `version 'OPENSSL_3.0.9' not found`, aborting the apt transaction and failing the isaac-sim image build (PX4 Tools/setup/ubuntu.sh, exit 100). Clear LD_LIBRARY_PATH for that RUN only so apt/openssl use the system libcrypto; the global ENV still applies to every other layer. Environmental break (new ca-certificates × isaac-sim's stale bundled openssl) — not a code regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…378) * Add waypoint_flight system test judged by standalone track checker New end-to-end acceptance test for planner integration/swaps: takeoff -> ordered waypoint route -> land, per (sim, num_robots, iter). - tests/system/test_waypoint_flight.py (mark: waypoint_flight): after takeoff, sends the route to the local planner's NavigateTask action as a nav_msgs/Path and captures odometry throughout; reuses the flight-cycle workers from test_fixed_trajectory.py (chain guard, takeoff/land, odom CSV capture). - tests/waypoint_checker.py: standalone stdlib-only judge — the odometry track must pass within --waypoint-tolerance of every waypoint IN ORDER, each within --waypoint-timeout of the previous arrival. Success is defined purely on the odometry track (not the action result), so swapping the global or local planner leaves the judgment unchanged; the checker also runs outside the harness on any ros2 `topic echo --csv` odometry dump. - Waypoints are relative to the robot pose at dispatch (x forward along heading, z up), so routes are spawn/sim agnostic. Default: 10 m square at takeoff altitude. - New pytest options: --waypoints, --waypoint-tolerance, --waypoint-timeout; mark registered in pytest.ini; docs in tests/README.md and AGENTS.md; VERSION 0.19.0-alpha.9 + CHANGELOG. Metrics recorded per robot: waypoint_success, waypoints_reached, navigate_action_success, route_time_sim_s, worst_closest_approach_m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Calibrate waypoint_flight to validated stock behavior in Isaac Sim Validated end-to-end against Isaac Sim + the stock stack (4/4 phases pass in 3m20s; corners cut 3.75/5.13 m, final goal error 0.63 m). Fixes found by flying: - Path header frame: an empty frame_id crashed droan_gl (uncaught tf2::InvalidArgumentException in its plan TF transform); the goal now carries the frame from the odometry snapshot (fallback "map"). - Dense plan dispatch: sparse poses get corner-skipped by the local planner's distance-walking look-ahead; the route is now interpolated at 1 m from the current pose (mirrors real global-planner output). - Route/tolerance semantics: the stack's contract is "reach the goal precisely, follow the corridor loosely" (droan_gl cost = deviation - path_distance cuts corners ~4-7 m). Split tolerances: intermediate corridor 15 m, final goal 2.5 m (new --goal-tolerance; NavigateTask's 1.5 m + tracking lag). Default route is now an open 30 m square — NavigateTask succeeds on distance to the FINAL pose, so closed loops succeed instantly without flying (documented). - Settle capture: the action succeeds on the tracking point, which leads the drone by up to the look-ahead distance (~10 m); capture now continues until the drone is stationary (max 30 s) so the goal approach is recorded. New metric: final_goal_error_m. - waypoint_checker: closest_approach now reports the true minimum over the remaining track instead of the tolerance-boundary crossing (arrival stays first-crossing, ordering semantics unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Raise default waypoint route +10m to clear scene clutter Validated on both sim backends with the identical default config (open 30 m square climbing to ~20 m AGL): - Isaac Sim: corners 5.67/5.72 m, final goal 0.28 m, 4/4 phases - ms-airsim (Blocks): corners 5.93/5.67 m, final goal 0.89 m, 4/4 At the old takeoff-altitude route the drone collided with a Blocks obstacle (disparity was streaming, so DROAN had perception — the corner-cut diagonals leave the forward stereo's coverage). This test judges route-following, not obstacle avoidance, so the default route flies above the clutter; documented in the option help and README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add waypoint_flight screenshots from validation runs Captured mid-route during the validated flights: Isaac Sim viewport with the drone on the square route, ms-airsim Blocks with the drone clearing the obstacle field (collision count 0), and the Foxglove GCS dashboard showing the planned path, expanded obstacle voxels, robot task panel, and live stereo feed. Embedded in the waypoint section of tests/README.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…s feeding PRs (#381) * Add feature-notebook workflow: local design specs + test results per feature Every feature a coding agent implements now gets a numbered entry under notebook/ (gitignored, local-only): a design_spec.md written before coding (problem context from the session, proposed implementation with per-section DESIGN/TODO / WIP / DONE status labels, lettered test plan) and a results/ tree with per-section raw artifacts plus a self-contained results_summary.md (embedded tables + figures) that populates the feature's PR description. - New skill .agents/skills/use-feature-notebook with SKILL.md and design_spec / results_summary templates - AGENTS.md: skill registry row, notebook-first Agent Workflow Example, new "Feature Notebook" section - .gitignore: /notebook/ Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Bump version to 0.19.0-alpha.10 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Document the feature notebook workflow under Development docs Adds docs/development/intermediate/feature_notebook.md (directory layout, 5-step workflow, status labels, local-only rule, notebook → PR flow), wires it into the mkdocs nav under Development > Intermediate Tutorials > Contributing, and lists it in the Development index. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…tity failure (#377) * make RECORD_BAGS actually reach the bag recorder LOG_CONFIG selects which topic set in logging_bringup/config to record, default log.yaml. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * warn when the robot identity fails to resolve Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: bump version to 0.19.0-alpha.12 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fixed comments and documentation * fix the bag recording status bridge direction It was bridged gcs -> robot, the same direction as the command it answers, so status never reached the GCS and the rqt Recording: label stayed blank. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix the exclude flag so the main bag section records ros2 bag record renamed --exclude to --exclude-regex, and the old name is now an ambiguous prefix of four options, so argparse rejected the command and any section using exclude: recorded nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * restore the bags .gitignore files #318 dropped robot/bags/.gitignore and gcs/bags/.gitignore while moving a dozen others; nothing has covered recorded bags since. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
) * ci(orchestrator): migrate ephemeral CI runners from OpenStack to NVIDIA OSMO Replace the OpenStack-Nova spawn/reap backend with OSMO workflow submission. The GitHub side is unchanged (self-hosted/airstack-ephemeral labels, single-use JIT runner tokens, same-repo fork guard) and the one-job-per-worker destroy-after model is preserved; only the spawn target moved from creating a Nova VM to submitting an OSMO workflow. orchestrator.py: submit/query/cancel/list via the osmo CLI, job_id -> workflow_id state, re-login-on-auth-failure, orphan sweep via osmo workflow list; drop floating-IP/boot-volume/placement/keypair/security-group logic. runner.Dockerfile + runner-entrypoint.sh + runner-workflow.yaml.j2: prebaked privileged docker-in-docker + GPU GitHub runner image/task (replaces cloud-init.yaml.j2). config.example.yaml, setup.sh, airstack-orchestrator.service, requirements.txt: OSMO service-account token auth, install the osmo CLI, drop openstacksdk. Docs (AGENTS.md, tests/README.md, orchestrator README) updated to OSMO. Co-authored-by: Cursor <cursoragent@cursor.com> * ci(orchestrator): pin AirLab OSMO JSON keys and runner image path Resolve uuid/live name after submit (OSMO returns name-only + suffix), default config to the Keycloak-backed airstack pool and Harbor runner image, and add scripts to build/push airstack-ci-runner on OSMO DinD. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(ci): document the OSMO-backed CI/CD pipeline Fills in the empty ci_cd.md stub with an end-to-end guide to how CI runs the full AirStack stack on ephemeral OSMO GPU pods: architecture and job lifecycle diagrams, runner pod anatomy, the three trigger paths, what each pytest mark catches, the metrics regression gate, the security model, and layer-by-layer troubleshooting. Adds the page to the mkdocs nav (it was previously unreachable) and cross-links it from tests/README.md and the testing index. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): repair Docker builds on OSMO ephemeral runners Every build_docker and build_packages test failed on the OSMO backend because the inner dockerd kept its data-root on the pod's overlayfs rootfs. Linux rejects a directory on overlayfs as an overlay upperdir, so image pulls still succeeded -- containerd unpacks layers with plain writes -- while every build step needing a real mount died with "mount source: overlay ... err: invalid argument", surfacing as unrelated-looking apt-get and WORKDIR failures. runner-entrypoint.sh now picks a storage backend by attempting a real overlay mount rather than trusting the filesystem type, preferring a loopback ext4 data-root (real overlay2, sparse, dies with the pod) and falling back to a pod-mounted filesystem, fuse-overlayfs, then vfs. vfs is a last resort only: it copies the whole filesystem per layer and would exhaust the storage request on the sim images. Also bumps the GitHub Actions runner to 2.336.0, since 2.334.0 stops being able to run jobs on 2026-08-10. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): seed PR Docker builds from a floating cache tag Versioned cache_from entries always miss on PRs because VERSION is forced up; add a stable cache_* tag published only by docker-build.yml so system tests can reuse layers without writing the shared cache. Co-authored-by: Cursor <cursoragent@cursor.com> * ci(docker-build): retag unchanged images on VERSION bump Skip full compose rebuilds when a service's content fingerprint matches the previous versioned image label; registry-retag instead and only rebuild services whose Docker inputs changed. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): parse quoted .env values before inline comments docker_image_plan was feeding NUM_ROBOTS with a trailing comment into compose config, which broke strconv.Atoi for deploy.replicas. Co-authored-by: Cursor <cursoragent@cursor.com> * ci(docker-build): build/push services sequentially Publish successful images even when a sibling (e.g. isaac-sim) fails, and still cosign whatever was retagged or pushed in the same run. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: bump VERSION to 0.19.0-alpha.8 for retag validation Seeded gcs/ms-airsim/robot images carry content-fingerprint labels; this bump should registry-retag those digests without rebuilding. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): unblock isaac-sim PX4 apt and robot colcon pytest Isaac's PX4 ubuntu.sh fails dpkg configure on the NVIDIA base; pre-fix ca-certificates, drop software-properties-common, and skip NuttX/Gazebo like ms-airsim. Pin pytest<8.1 and disable launch_testing for colcon unit tests so ROS Jazzy's outdated pytest hook no longer aborts CI. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): pass colcon --pytest-args as separate tokens A single quoted blob made pytest treat "-p no:launch_testing" as part of the -m expression, which broke lidar_point_cloud_filter colcon tests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): quote colcon pytest args through bash -ic Nested single quotes around 'not linter' terminated the outer bash -ic string early, so pytest saw 'not' as a path. Use shlex.quote for the whole command and list-form pytest_args in the YAML. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): pass colcon pytest flags via PYTEST_ADDOPTS colcon --pytest-args is a single nargs='*' option, so repeating it dropped -p and pytest treated no:launch_testing as a file path. Set PYTEST_ADDOPTS with docker exec -e instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): rename helper so pytest does not treat it as a hook conftest functions named pytest_* are registered as hooks. pytest_addopts_env caused PluginValidationError and exit code 3. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: skip image-build for build_packages reruns Pull and retag cache_* images instead of baking isaac/airsim on every colcon/pytest iteration. /pytest --no-image-build does the same for other marks. compose up --no-build when AIRSTACK_NO_IMAGE_BUILD=1. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): disable pytest plugin autoload for colcon tests -p no:launch_testing is applied after setuptools entrypoints load, so pytest 8.1+ still crashes on launch_testing's path= hook. Set PYTEST_DISABLE_PLUGIN_AUTOLOAD so cache_* robot images (unpinned pytest) can run lidar tests without a rebuild. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): skip lidar ament linters in package pytest config PYTEST_ADDOPTS -m not linter never reached ament pytest, so copyright / flake8 / pep257 still ran after the unit tests passed. Ignore those modules in setup.cfg and collect_ignore. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: default system tests to isaacsim only PR-open and bare /pytest were sweeping both sims. Default --sim to isaacsim; msairsim is opt-in via --sim msairsim. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: pvkumara <pkumara@andrew.cmu.edu> Co-authored-by: Cursor <cursoragent@cursor.com>
#374) * feat(perception): bring natnet_ros2 client up to the optitrack_emulation baseline Take the natnet_ros2 package from #367 onto the reworked base: the C++ NatNet client (natnet_ros2_node + client adapter + natnet_logic seam), the base mavros_gp_origin and vision_pose_converter nodes, per-robot natnet_config profiles, launch files, and the co-located C++/Python unit tests. natnet_ros2 is already listed in tests/colcon_unit_test_packages.yaml, so the base's YAML-driven collection picks up the updated unit tests directly — no proxy files. Real-robot PX4 external-vision fusion (px4_param_setter, geoid-corrected origin, EV-pose bounds) is layered on next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(natnet): real-robot PX4 external-vision fusion (mocap → EKF2) Layer the Hummingbird real-robot fusion pipeline onto natnet_ros2 so an OptiTrack-only drone (no GNSS/mag/baro) fuses mocap pose into PX4 EKF2: - mavros_gp_origin_node: publishes a guarded synthetic GPS origin. On real HW, use_geoid_altitude feeds the egm96-5 geoid undulation (N ≈ 54 m at Lisbon) so mavros's ellipsoidal→AMSL conversion cancels and local z == OptiTrack z (fixes the ~36 m = 90 − 54 boot offset; see docs). Auto-skipped in sim. - vision_pose_converter_node: rate-limited mocap → MAVROS vision_pose bridge. - px4_params.yaml: the external-vision EKF2 param set. - natnet_ros2.launch.py wires the bridges when a robot's vision_pose block is on. px4_param_setter reworked into a **checker** (R3): auto_set=false by default — it reads and *flags* FCU params that differ from the desired set instead of writing them; on_mismatch=warn|halt (default warn). Set the params in QGroundControl; the node is the pre-flight safety net. auto_set=true restores the legacy enforce path. Excludes the duplicate vendored NatNet SDK (sensors/natnet_ros2) and deployment override .envs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(natnet): PX4 external-vision setup guide + height-datum explainer Move the PX4 external-vision setup guide into docs/ (was a repo-root markdown) and wire it into the mkdocs nav under Perception. Adapt it to the reworked param checker (auto_set default off; check-and-flag, not enforce), and add a "height datum" section explaining the ~36 m local_z offset: AirStack's 90.0 ellipsoidal world datum minus the egm96-5 geoid undulation (N ≈ 54 m at Lisbon) = 36 m; fixed by publishing the geoid-corrected origin altitude so mavros's conversion cancels. Documents why it's invisible in sim and why the shared 90.0 datum must not be changed globally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(perception): point natnet launch include at the natnet_config schema Refine the perception bringup comment on the LAUNCH_NATNET include so it points at the per-robot natnet_config.yaml schema parsed by natnet_ros2.launch.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: bump version to 0.19.0-alpha.14 * fix(natnet): make the NatNet client actually reachable + correct EV tuning Three defects that together meant the OptiTrack client could never connect to anything, in sim or on a real robot. 1. NATNET_SERVER_IP was unreachable config. natnet_config.yaml resolves it via $(env ...), but docker compose only injects variables named in a service's `environment:` block and no service declared it — not the compose files, not .env, not tests/system/test_optitrack_e2e.py. The client therefore always fell back to its hardcoded default (192.168.123.199), which is neither the in-sim emulator (172.31.0.200) nor any Motive host. Forwarded in robot-base-docker-compose.yaml, defaulting to the emulator so the sim path works unconfigured. 2. The tracked rigid body could never match. robot_1 pinned "Hummingbird" id 1146 while the emulator streams "Drone" id 1, and the NatNet client filters incoming frames by NUMERIC id — a mismatch yields a connected client that silently never publishes. Body name/id now accept $(env ...) (expanded in _build_node_params, with the id still coerced to int) and default to the emulator's body; sites override via NATNET_BODY_NAME / NATNET_BODY_ID. 3. EV tuning was not the deployment-validated set. EKF2_EV_DELAY 8.0 -> 7.0 and EKF2_EVP_NOISE 0.01 -> 0.05. EKF2_EVP_NOISE is not marker precision: it also sets the innovation gate at EKF2_EVP_GATE (default 5) sigma, so 0.01 gave a 5 cm gate that rejected legitimate mocap updates and refused to arm. 0.05 is a 25 cm gate, still far tighter than PX4's 0.1 default. px4_params.yaml keeps the evidence inline, including two results that are expensive to rediscover: raising EKF2_EV_DELAY to 50.0 measurably degrades tracking (the negative best-fit time shift shows the estimate running ahead of truth), and the drift-and-snap excursions were a 90 deg body-yaw offset in the Motive rigid-body definition, not a gate problem — so the fix belongs in Motive, never as yaw compensation in code. Adds two unit tests covering body-field env expansion and the emulator-matching defaults (natnet_ros2: 14 -> 16 passing). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * add a real-robot OptiTrack deployment override Mocap counterpart to l4t-px4-realrobot.env: same Jetson stack, plus the NatNet server/body settings and LAUNCH_NATNET. Carries the two things that are easy to get wrong and produce no error. The body id must match Motive's streaming id, since the client filters frames numerically and a mismatch just never publishes. And nothing writes the EKF2 external-vision parameters to a real FCU — px4_param_setter only reads them back and warns — so they have to be set once in QGroundControl. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * config bodies per robot profile; trim comments to the docs The rigid body a robot tracks is now set only in its natnet_config.yaml profile, keyed by ROBOT_NAME. NATNET_BODY_NAME / NATNET_BODY_ID are gone: a single global env var cannot express per-robot values, so it blocked the multi-robot case the profiles already handle. NATNET_SERVER_IP stays in the environment — one Motive host serves every robot. Comments across the package are cut back to what is not evident from the code. The EKF2 tuning results that were buried in px4_params.yaml move into docs/robot/px4_external_vision.md, which also had stale values (EV_DELAY 15.0, EVP_NOISE 0.01) contradicting the config: that raising EV_DELAY measurably hurts tracking, and that drift-and-snap was a Motive rigid-body yaw offset rather than a gate problem. Kept: the license header, and the note on why the SDK needs a reachability pre-check before Connect(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * put the mocap floor at the shared world datum desired_floor_amsl 0.0 -> 36.0, the world datum (90 m ellipsoidal) expressed in AMSL, so a mocap robot's reported global altitude agrees with sim and the GCS instead of sitting at sea level. The published ellipsoidal origin works out to ~90 m, the datum itself. local_position.z equals the OptiTrack height for any value of this parameter — it only moves the global altitude. Reasoning lives in the external-vision doc, which also now records that GeoPoint.altitude is ellipsoidal by contract, so AMSL must not be sent here. Not yet confirmed on hardware. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fail the build when the geoid dataset is missing MAVROS constructs the egm96-5 geoid in its UAS core, before any plugin loads, and throws std::invalid_argument if the dataset is absent — mavros_node terminates at startup, so there is no MAVROS at all, GPS or mocap. The image could ship without it. mavros' install_geographiclib_datasets.sh sends the downloader's output to /dev/null and, on failure, prints "Error while installing" and returns without a non-zero exit, so the RUN layer succeeded regardless. The tool it calls, geographiclib-get-geoids, was also only a transitive dependency of ros-mavros rather than something we pinned. Now pins geographiclib-tools and asserts the file landed, so a failed download fails the build. Verified against the shipped image: with the downloader broken the script still exits 0, and the new test -f returns non-zero. This is the dependency the OptiTrack external-vision path needs — mavros_gp_origin resolves the geoid undulation with the same egm96-5 model — hence landing it here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * abbreviated Dockerfile comment on geographic lib installation * fix repo-root doc links in the external-vision guide They resolved relative to docs/robot/, so mkdocs looked for docs/robot/robot/ros_ws/... and warned on every one. Prefixed with ../../; the file now builds warning-free. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * comment trim * point the companion-link section at the PX4 docs Section 3 documented MAVLink serial setup at length — MAV_n_CONFIG / SER_TEL2_BAUD tables, wiring, USB-vs-TELEM2 comparison — all of which is standard PX4 setup that PX4 documents better and keeps current. Replaced with links to the companion computer, MAVLink peripherals, and serial configuration pages. Kept the part PX4 does not cover: the Cube Orange USB CDC-ACM stall, which starves EKF2 of vision updates and is why the companion link belongs on TELEM2. Four other sections and the troubleshooting table point here for that symptom. 65 lines -> 19. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * frame section 4 around mavros_gp_origin, demote the 36 m note Section 4 now leads with what mavros_gp_origin does — inject a synthetic global position so PX4 will arm in modes that need one without GNSS — rather than presenting the height datum as a peer topic. The ~36 m offset becomes a note under it, scoped to real deployments and ending with why sim never sees it (the geoid path is skipped under use_sim_time, and sim's synthetic GPS is self-consistent with the spawn). Section 4b is gone; it had no inbound references. Dropped the "don't change the 90.0 globally" warning. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * reject an unknown connection_type instead of defaulting to unicast validate_connection_type returned "unicast" for anything it did not recognise, so "mutlicast" or "Unicast" produced a client that connected on the wrong transport and then never received a frame — with only a warning to show for it. It now throws std::invalid_argument naming the offending value, and the node turns that into a fatal startup error rather than a warning it flies past. Case-sensitivity is deliberate: accepting "Unicast" would mean the config silently disagrees with itself. Tests updated from fallback to throw, plus one asserting the message names the bad value. 60 gtests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * px4 external vision docs trim * trim natnet node comments; note the latency figure is an estimate Comment trims in natnet_ros2_node.cpp (no code change). Records what cube_orange_latency_ms actually is: an estimate of the FCU hop, added to a logged total and never fused. Only the transport half of EKF2_EV_DELAY is measured, and that measurement starts at the NatNet server transmit, so Motive's own capture pipeline is not in it either. Also notes, for whoever retunes next, that the node stamps poses with its receive time — so delay after that stamp does not belong in EKF2_EV_DELAY, which points lower than 7.0 and matches the negative best-fit shift already recorded. Not chased down; 7.0 flies. CameraMidExposureTimestamp would replace the estimate with a measurement if it ever matters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * trim the external-vision tuning notes Replaces the two long tuning write-ups with a short troubleshooting tip (check the Motive rigid-body definition first — x forward, z up) and cuts the latency section back to what is measured versus estimated. Fixed a dangling "see below" in the EKF2_EV_DELAY table row, which pointed at the removed tuning result; the warning it carried is now stated inline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(sim): add NatNet server emulator (protocol core) + register unit tests The pure-Python NatNet server that emulates an OptiTrack Motive server so natnet_ros2 can be driven without hardware. USD/Isaac-free — this is the protocol + server core (unicast server, data/model/server types, serializers, default catalogs). The Isaac wrapper that maps a USD scene onto this server lands next. Registers the emulator package's co-located unit tests via a `sim:` entry in tests/colcon_unit_test_packages.yaml (base's simulation/**/<pkg>/test glob). The root conftest now puts each unit-test package's import root on sys.path so co-located tests import their package without a per-package conftest.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(natnet): host integration tests — emulator server → natnet_ros2 Drive the real natnet_ros2 client from the host NatNet server emulator and check the drone pose reaches ROS at rate (single-body and multi-body profiles). No sim, no GPU — uses the base's `robot_autonomy_stack` fixture + `integration` mark. The Isaac-wrapper variant lands with the Isaac wrapper PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: bump version to 0.19.0-alpha.15 * pack frame sections through one helper * fix the labeled-marker struct format that raised on every pack sMarker.pack used '<i5fhf', which expects five floats between ID and params but is only given four (x, y, z, size), so every call raised struct.error. Nothing hit it because the emulator streams rigid bodies only, leaving nLabeledMarkers at 0 and the section never packed. * ignore the editable-install egg-info in the emulator extension * comment trim * put helper-module dirs on sys.path for co-located unit tests The emulator's tests import natnet_test_helpers from their own test/ dir, which only resolved because test_natnet_integration.py inserts that path at import time and pytest imports it during collection. Narrowing the run (-m unit --ignore=integration) dropped that side effect and broke collection. The dir is added only when it ships no conftest.py, so lidar_point_cloud_filter keeps its parent-only path — putting its test/ dir on sys.path would shadow this conftest as module `conftest` and break every `from conftest import`. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…trajectory e2e (#376) * feat(sim): Isaac wrapper for the NatNet emulator (USD scene → server) The Isaac integration layer that maps a live USD scene onto the NatNet server: catalog/config/frames/manager/scene_setup/ui_extension/usd_bindings, the extension manifest (config/), and the USD schema. Adds the natnet Pegasus launch scripts that spawn the emulator alongside PX4 in Isaac Sim, the isaac unit tests (incl. a float-tolerance loosen on the pose round-trip for float32/USD noise), and the Isaac-wrapper host integration test. scipy + usd-core added for the emulator's USD/pose-sampling tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(natnet): dedicated OptiTrack sim e2e (optitrack mark) One dedicated Isaac bring-up (example_one_px4_pegasus_natnet_launch_script + LAUNCH_NATNET=true) that asserts the full NatNet chain: emulator → natnet_ros2 pose_cov >= 5 Hz, then PX4 local_position alive (EKF2 fusing the vision). Its own `optitrack` mark + _MODULE_ORDER slot — deliberately NOT a third parametrized sim, so the generic liveliness/sensors/flight suites aren't re-run under NatNet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(natnet): emulator sim doc + optitrack-development skill Add the NatNet emulator Isaac Sim documentation (docs/simulation/isaac_sim/ natnet_emulator.md) and the optitrack-development agent skill covering the emulator, natnet_ros2, and the NatNet wire-protocol handshake. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: bump version to 0.19.0-alpha.16 * fix(sim): register the NatNet emulator via the Kit ext-folder The Isaac launch scripts import `optitrack.natnet.emulator`, but Kit was only pointed at the shared exts dir (`~/.local/share/ov/data/documents/Kit/shared/exts`), where Dockerfile.isaac-ros installs pegasus.simulator at image build. The emulator lives in the repo at simulation/isaac-sim/extensions/ and is never copied there, so it was not a registered extension and the import depended on ambient sys.path. Kit accepts repeated --ext-folder, so both standalone commands now pass the repo's extensions dir as a second search root. Chosen over copying the extension into the shared dir at build time because the repo tree is bind-mounted: emulator edits take effect on relaunch instead of requiring an image rebuild. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * make the sim actually fuse the mocap stream EKF2_EV_CTRL defaults to 0, and the isaac compose set no PX4 params at all, so PX4 discarded the vision entirely and flew on sim GPS. The emulator could stream perfectly and change nothing. PX4 SITL's rcS applies any PX4_PARAM_<NAME> env var at boot and Pegasus passes the container env through, so no new mechanism is needed. Each entry defaults to PX4's own default, read out of the firmware in this image — unset is an explicit no-op and non-mocap sims are unaffected. They cannot be defined-but-empty: the rcS loop has no empty-value guard. Also hooks NATNET_BODY_ID in the single-drone launch script. The emulator hardcoded streaming id 1 while the client reads the env var, so a real Motive id would desync the two into a connected client that never publishes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fly a circle on mocap fusion instead of asserting a topic exists test_px4_fuses_vision claimed to prove EKF2 fused the external vision but only waited for local_position/pose, which publishes off GPS regardless — it passed with vision disabled. The stack now comes up with GPS, baro and range aiding off, so mocap is the vehicle's only position source, and the module flies the Circle trajectory. Sustained lateral motion is where a wrong EV delay or a too-tight innovation gate shows up; a hover would not reveal either. Cross-track error is scored by the same helpers the autonomy benchmark uses, imported rather than reimplemented. test_px4_fuses_vision is kept as the pre-flight gate — it now establishes only that an estimate exists, and says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * enforce only the mocap circle flight on PR open The pull_request branch passed no args, so opening a PR ran pytest's defaults: every mark, both sims, all four trajectory types. Now it runs the one end-to-end flight that covers the whole chain. Every other suite is unchanged and still reachable on demand — /pytest comments, workflow_dispatch inputs, and local airstack test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * add an isaac natnet mocap override Brings up the emulator plus PX4 on external-vision fusion in one command — the same configuration test_optitrack_e2e.py uses, so the test environment is reproducible by hand. Sets PLAY_SIM_ON_START explicitly because the root .env ships it false: the scene then loads paused, /clock never ticks, and every use_sim_time node sits frozen while the stack looks healthy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * install the natnet emulator as a real Kit extension The natnet launch scripts died with ModuleNotFoundError: No module named 'optitrack'. Pointing Kit's --ext-folder at the repo extensions dir was not enough — that only makes Kit aware of an extension, it does not put the package on sys.path. Handle it the same way pegasus.simulator already is: bake a copy into the Kit shared exts dir and pip-install it editable, then bind-mount the repo copy over it so edits stay live. The scripts now enable_extension() before importing, which registers the extension and its omni.isaac.core / omni.usd dependencies. The repo-extensions --ext-folder flag is dropped; the extension now lives in the dir the image already searches. Verified in a running container: extension starts, emulator serves on 172.31.0.200 :1510/:1511, and the robot sees /robot_1/perception/optitrack/drone/pose_cov at ~101 Hz feeding vision_pose and PX4 local_position at ~32 Hz. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * set the streamed body in the script, not the environment The emulator read NATNET_BODY_NAME / NATNET_BODY_ID from the environment to stay in sync with the client. The client now takes its bodies from its per-robot profile in natnet_config.yaml, so the env hook was asymmetric and, being global, could not describe a multi-robot scene anyway. Both are now constants in the launch scripts, with the pairing spelled out inline, in the emulator sim doc, and in the optitrack-development skill — including that a mismatched id fails silently: the client connects and never publishes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * comment trim on isaac-sim docker compose * point the isaac-sim env blocks at their documentation * comment trim on editable installation of natnet emulator * keep the full default test run on PR open Narrowing the PR gate to `-m 'build_packages or optitrack'` also dropped the unit tier — 155 tests, including the emulator's own suite, which colcon test does not cover (it runs only the robot workspace packages). The optitrack e2e needs no gate of its own: with no -m filter it is collected like everything else, and it brings up its own mocap-EV stack via _E2E_ENV. Only the heavy-mark classification stays, so /pytest -m optitrack still builds sim images instead of taking the pull-only path. * wait for a converged estimate before arming in the optitrack e2e Gate on local_position/odom instead of /pose. odom goes live only once EKF2 has converged and home is set, which is what PX4's arming preflight requires; /pose fires earlier, and the takeoff dispatched in that window returned "failed to arm". Both autonomy suites already gate on odom for this reason (test_px4_ready). The gate alone is not sufficient under external vision: with GPS, baro and range aiding off, PX4's heading and horizontal-position stability checks settle after odom starts publishing — measured at ~26s past the gate. TakeoffTask does not retry its own ARM, so retry here first. * comment trim on optitrack e2e collection ordering * comment trim on the PR-open test args * rename the isaac natnet override to isaac-optitrack-simulation.env * select PX4 SITL parameters with a named env_file The isaac-sim service listed eleven PX4_PARAM_* entries, each defaulting to a hardcoded copy of PX4's own default so that an unset value stayed a no-op — rcS has no empty-value guard. Those copies can drift from firmware silently. Replaced with env_file: ./px4-params/${PX4_PARAM_SET:-default}.env. default.env is empty, so an unselected run injects nothing and PX4 keeps its firmware defaults; external-vision.env holds the mocap set. An unknown name fails the compose config rather than falling back. Also corrects the natnet_emulator doc table, which described three robots, the multi-drone script, and a SITL_PARAM_PROFILE variable that exists nowhere. * comment trim in compose file * trim verbose comments in the natnet sources Shorten multi-line inline comments that explained rationale or compared the chosen approach against alternatives. The longer explanations already live in docs/simulation/isaac_sim/natnet_emulator.md, so the comments now state what the code does and point there. Limited to files this PR adds: the natnet launch scripts and the emulator's isaac/ modules. The env files and the pre-existing launch script keep their original comments. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * removed comment change * drop the GPS origin change from the baseline pegasus launch script example_one_px4_pegasus_launch_script.py is a pre-existing non-mocap script and does not need to change for the NatNet emulator work, so restore it to develop. The set_gps_origins call was also inert here: for a single drone spawned at the world origin it computes (38.736832, -9.137977, 90.07), which is the Lisbon default gps_utils already documents, and nothing in the Pegasus submodule reads the PX4_HOME_LAT_<domain_id> vars it writes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * assert the external-vision params actually reached the FCU The rest of this module assumes PX4_PARAM_SET=external-vision took effect. If it silently does not, EKF2_EV_CTRL stays 0 and EKF2_GPS_CTRL stays 7, the vehicle flies the Circle on sim GPS, and every test still passes — the proof-by-elimination in test_px4_fuses_vision collapses because the elimination never happened. Read EKF2_EV_CTRL and EKF2_GPS_CTRL back off the FCU through the MAVROS param plugin, so the check covers the whole chain: compose env_file -> container env -> Pegasus -> PX4 rcS -> FCU. Runs before the flight tests so a param failure short-circuits in seconds instead of after two 2400s timeouts. Two params, not the full set: if these are right, PX4_PARAM_SET demonstrably applied and the rest came with it. Matching is on the printed value line, not the exit code — an unpulled param prints "Parameter not set." and still exits 0. Verified against a live sim: passes on the real config, fails with distinct messages for a wrong value and for a param that never appears. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(natnet): publish the emulator page and correct the setup examples Add the emulator doc to the nav as "MoCap Emulator" — it built and served but was orphaned, so it was only reachable by knowing the URL, and the "See docs/..." pointers in the code led somewhere unnavigable. Fix the launch-script examples in the doc and the extension README. Both omitted enable_extension(), which is the actual prerequisite: the package imports fine because Dockerfile.isaac-ros pip-installs it, but the emulator's modules pull omni.usd / omni.physx lazily, so Kit has to have the extension registered. The doc also carried a sys.path.insert pointing at ../utils (where scene_prep lives) that had nothing to do with the optitrack import. The README targeted /World/drone1/base_link rather than the /body child the launch scripts stream. Document that client registration does not survive a server restart: restart the robot container after Stop/Start Server. Stopping and starting the simulation is unaffected — frames are sampled on the physics step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(natnet): the extension owns the server, tied to the sim timeline The Kit extension is the single owner of the NatNet server. It builds one from the /World/NatNetInterface prim on Play and shuts it down on Stop, so the server's lifetime matches the simulation and the panel reports its state rather than controlling it. Launch scripts author the interface prim before starting the timeline; author_drone_natnet_interface writes the prim and returns the authored config. Because the server is constructed on each Play, serverIp/ports/mode — bound into the socket at construction — pick up whatever is authored at that point. Bodies, up-axis and pose noise are re-read while running and need no rebuild. The panel opens on the interface authored on the stage, so Save writes back what is there; author_interface replaces the whole body set. A client registers with the server instance it connects to, and natnet_ros2 handshakes only until its first success, so a client from an earlier run is unknown to the server built by the next Play. Restart the robot container after each Stop -> Play cycle; documented in natnet_emulator.md. Not exercised against a live panel yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs trim --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* docs(tests): align unit-test docs with the co-located layout Unit test source moved into <package>/test/ and is collected from colcon_unit_test_packages.yaml, but the surrounding documentation still described the mirror-directory-and-proxy scheme that replaced. Six per-layer stubs under tests/robot/ told authors to add tests in directories tests no longer live in, and tests/sim/motive_emulator/README.md proposed a NatNet emulator that was built at simulation/isaac-sim/extensions/optitrack.natnet.emulator/ instead. Remove them and rewrite the two tree READMEs as signposts. Correct the add-unit-tests and run-system-tests skills, which future agents read to work in this area, on four points they had wrong: - Running them. `pytest tests/` does not collect co-located unit tests — the injection in conftest.pytest_configure is skipped whenever a path is given on the command line. It reports "no tests collected" and exits 5, which reads as a failure but means nothing ran. `airstack test -m unit` and `cd tests && pytest -m unit` are the working forms; verified 155 passed vs exit 5. - CI. No workflow runs unit tests. system-tests.yml invokes `pytest tests/`, and fires only on PR-open, /pytest, or workflow_dispatch. - The mark. pytest_itemcollected applies @pytest.mark.unit by file location, so test sources should not declare it. The skill previously said "always decorate", which is where the redundant declarations came from. - colcon. It runs only what a package's CMakeLists registers. natnet_ros2 has ament_add_gtest but no ament_add_pytest_test, so its Python tests run only under the root harness. Also fixes a pytest_args example that would silently do nothing (`-m not linter`; ament's pytest runner ignores -m via PYTEST_ADDOPTS, and the real value is []), and the same stale layout claim in the testing docs and the emulator README. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(tests): record how C++ and Python unit tests reach CI C++ gtests run under colcon test, which CI executes inside the robot container via the build_packages mark (test_build_packages.py::test_colcon_test_robot). Python unit tests run under the root harness, which no workflow invokes. Whether colcon test also picks up a package's Python tests depends on its build type: lidar_point_cloud_filter is ament_python and exposes them via setup.cfg (testpaths = test), so they run in both places; natnet_ros2 is ament_cmake and registers only ament_add_gtest, so its Python tests run nowhere in CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(tests): collect co-located unit tests when the run is not narrowed Unit-test source lives outside tests/, so pytest_configure appends it to the collection args. That injection was gated on args_source != ARGS, which pytest sets for any positional path — including `tests/`. The intent was that `pytest tests/system/foo.py` should not drag in 155 unrelated tests, but the guard could not tell narrowing from naming the whole suite, so CI's `pytest tests/` collected 97 of 252 items and the Python unit tests ran nowhere. Decide on the paths instead: a positional is broad when it names tests/ itself or an ancestor, narrow otherwise. `pytest tests/` and `pytest .` inject; `pytest tests/system`, a single file, and a node id do not. Node ids are split on `::` first, since only the part before it addresses the filesystem. `any` rather than `all` is deliberate — pytest_configure appends the co-located files (narrow, absolute) to config.args, so `all` would flip the answer for anything re-deriving it after that mutation. The decision is also stashed on config for the contract test to read. tests/meta/test_collection_contract.py pins the behaviour: a table over broad/narrow invocations, a check that the command in system-tests.yml is classified broad (the test that would have caught this), and a check that every discovered file produced collected items. It lives under tests/ on purpose — co-located, it would stop being collected at the same moment it stopped guarding anything. Verified: `pytest tests/ -m unit` 0 -> 170 passed; `cd tests && pytest -m unit` unchanged at 170; `pytest tests/system/test_liveliness.py` still collects 16. Unit tests now run with every system-tests.yml invocation. That workflow's triggers are unchanged and intentional — PR open, /pytest, workflow_dispatch — since the same run drives the GPU system tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(tests): explain why C++ and Python unit tests use different runners The split was documented as a fact without its reason. A gtest is a binary compiled against the package's headers and rclcpp, so it can only run where the ROS toolchain is — colcon test inside the robot container, which build_packages reaches after building with -DBUILD_TESTING=ON. Python unit tests stub ROS at the import boundary and touch no ROS runtime, so they need neither a build nor a container, which is what keeps the suite under a second. State the invariant that follows: a Python test needing a live ROS node belongs in tests/integration/ or tests/system/, not in a package test/ dir. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: run the collection contract tests with the fast tier They are hermetic and they guard the collection of everything above them, so running them after the GPU sim suites is backwards — a hung flight test would mean they never execute. Rank them in _MODULE_ORDER right after the co-located unit tests, ahead of system.test_build_docker. Also drop the `from conftest import repo_path` in favour of harness.discovery, which the module already imports from — one less thing between the test and the function it needs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): make PR validation and metrics trustworthy Run fast unit checks automatically, constrain host collection, and distinguish infrastructure failures from comparable simulation results. --------- Co-authored-by: John <johnliuchs2022@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Pranav Kumara <pkumara@andrew.cmu.edu>
Add a 'date and timestamp everything' convention to the use-feature-notebook skill: Date started / Last updated in design_spec.md, run timestamps on stored test artifacts, and per-section run times in results_summary.md. Update both templates accordingly and add a pitfall for undated documents. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h-script dedup, truthful logs (#386) * refactor(isaac): dedupe launch scripts into shared PegasusApp base The six launch scripts were 80-90% copy-pasted boilerplate (extension enabling, wait_for_stage, scene prep, spawn calls, run loop) that had already drifted: livestream existed only in the *_one_* scripts (so the isaac-sim-livestream service silently black-screened with multi scripts), ISAAC_SIM_HEADLESS was honored only by the *_multi_* scripts, and barebones_pegasus_launch.py (the documented template) crashed with a NameError (os never imported). pegasus_app.py now owns the skeleton once: create_simulation_app() (livestream + headless env handling, uniform across all scripts), extension enabling, world/env loading, scene prep, drone/sensor spawning from config dicts, and the run loop. Scripts reduce to scenario declarations plus hooks (pre_scene_prep/post_scene_prep/post_spawn). Behavior preserved per script (spawn poses, prim/node names, sensor offsets, NatNet bodies, GPS origins), with three deliberate fixes: - ISAAC_SIM_HEADLESS and ISAAC_SIM_LIVESTREAM now work in every script - barebones template runs again - NATNET_BODY_NAME/NATNET_TARGET_NAME env overrides now work as the one-drone natnet script's docstring already claimed example_multi_drone_scene_import keeps its historical ZED offset [0.21, 0, 0.05] (drift vs the canonical [0.2, 0, -0.05] — now visible and annotated instead of buried). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(cli): intent flags on 'up', resolved-value preflight, and 'airstack ready' airstack up learns intent flags that derive the coordinated env-var sets users previously had to know by heart (they export leaf values only — compose interpolation gives shell env precedence, so .env is untouched): --sim isaac|airsim swap simulator profile + matching URDF --robots N NUM_ROBOTS + auto-select one/multi Isaac script (also natnet pair; warns on custom scripts) --headless ISAAC_SIM_HEADLESS + MS_AIRSIM_HEADLESS + QT offscreen --play/--no-play PLAY_SIM_ON_START --no-autolaunch AUTOLAUNCH=false --wait chain into 'airstack ready' after compose up --dry-run print + validate the resolved config, start nothing Every up prints the resolved launch config and dumps it to .airstack/runs/<ts>/effective_config.env (gitignored; best-effort on read-only checkouts). Preflight now validates RESOLVED values (env > --env-file > .env), fixing the historical guard bypass where 'up --env-file overrides/...' was checked against .env only. New checks: NUM_ROBOTS>1 with the single-drone Isaac script (previously a silent 3-containers-1-drone failure) is a hard error; missing images are listed by name with an image-pull hint before compose starts a multi-GB implicit build; missing omni_pass.env / empty Pegasus submodule / docker<29 name-resolution are surfaced on the host instead of dying invisibly inside tmux. AIRSTACK_SKIP_PREFLIGHT=1 downgrades errors to warnings. 'airstack ready' (and 'up --wait') answers "can I press Takeoff yet?": staged gates mirroring the system-test budgets — containers (120s) → sim /clock (600s) → per-robot sentinel nodes (300s) → PX4 MAVROS connected + local_position/odom streaming (300s, the EKF-converged armable signal; connected alone fires ~25s early). --json for scripts; per-gate failures name the container/tmux window to inspect. tests/meta/test_launch_intent_contract.py pins the flag derivations, guard behavior, and exit codes (runs under the unit mark). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(docker): tee tmux pane output to container stdout Every service runs its real workload inside tmux, so 'docker logs' / 'airstack logs' were empty by construction — colcon build failures, Pegasus import errors, and scene downloads all landed in panes nobody attaches to. tmux hooks in the shared .tmux.conf (mounted into robot, gcs, isaac-sim, and ms-airsim containers) now pipe-pane every created session/window/split to /proc/1/fd/1, making container logs truthful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: fix launch-workflow drift against actual code behavior Corrects statements the audit found wrong, and teaches the new flags: - getting_started: sim comes up PAUSED by default (PLAY_SIM_ON_START=false in .env, docs claimed auto-play), operator UI is Foxglove not RViz (DEBUG_RVIZ=false by default), adds 'airstack ready' / --wait and --sim/--robots variants - simulation index + isaac docker.md + key_concepts + docker_usage: ISAAC_SIM_SCENE does not exist — scene selection is ISAAC_SIM_SCRIPT_NAME (standalone) or ISAAC_SIM_GUI (USD path, non- standalone); defaults table now matches .env/compose (AUTOLAUNCH=true, PLAY_SIM_ON_START=false, ISAAC_SIM_USE_STANDALONE=true, 100 Hz physics) - simulation index: NUM_ROBOTS=3 alone does NOT put 3 drones in Isaac — documents --robots (auto script switch) and the preflight guard - docker_usage: the test service is robot-test, not autotest - gcs user_interface: gcs service is not in the deploy profile (gcs-real is) - ms-airsim: MAVROS connects on 14540+domain (24540+i is AirSim's own PX4 channel), camera FOV default is 90 not 110, vehicles are robot_<i> not drone<i> - AGENTS.md: airstack stop/build are not registered commands (down / image-build); documents the new up flags and ready - .env: correct usage comment; PLAY_SIM_ON_START paused-by-default note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(release): bump VERSION to 0.19.0-alpha.18 and update CHANGELOG Image inputs are unchanged (all edits are bind-mounted or host-side), so docker-build should registry-retag rather than rebuild on merge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(sim): document PegasusApp launch-script authoring; re-teach stale skills spawning_drones.md now documents the pegasus_app.PegasusApp base class as the way to write a launch script: import-order contract, constructor kwargs, the drone-config dict (incl. prim/node_name/sensor overrides), hooks (pre_scene_prep/post_scene_prep/post_spawn), and which reference subclass to study for scene-import and NatNet scenarios. pegasus_scene_setup.md points at it and drops the false 'PLAY_SIM_ON_START not supported in standalone mode' claim. docker_usage.md gains a 'Launch flags and readiness' section (--sim/--robots/--headless/--play/--wait/ --dry-run, effective-config dumps, airstack ready). The write-isaac-sim-scene skill was re-taught from scratch: it prescribed copy-pasting a ~240-line skeleton whose API had drifted to non-runnable (wrong add_zed_stereo_camera_subgraph signature, nonexistent SIMULATION_ENVIRONMENTS keys, low-level Multirotor API no shipped script uses). It now teaches scenario declaration on PegasusApp with an explicit 'do not copy-paste' rule. Other skills fixed where the old guidance became wrong or footgun-inducing: integrate-module-into-layer ('airstack stop' is not a command), test-in-simulation and configure-multi-robot (bare NUM_ROBOTS=N up now fails preflight with the single-drone script — use --robots), use-airstack-cli (new flags + ready in the reference), optitrack-development (single-drone NatNet body names are env-overridable now). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Promote the 0.19 series (intent-flag launch workflow, airstack ready, resolved-config preflight, OSMO ephemeral CI runners, OptiTrack external-vision configurations, feature-notebook workflow) out of pre-release: VERSION 0.19.0-alpha.18 -> 0.19.0; CHANGELOG [Unreleased] promoted to [0.19.0] - 2026-08-22. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Contributor
Test Metrics —
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ships the 0.19 series to main: intent-flag launch workflow (
airstack up --sim ... --robots N),airstack readyflight-readiness gates, resolved-config preflight validation, OSMO-backed ephemeral CI GPU runners, OptiTrack external-vision configurations (sim + real-robot), tmux→docker-logs mirroring, the feature-notebook workflow, and the fixes recorded in CHANGELOG[0.19.0] - 2026-08-22.On merge:
docker-build.ymlbuilds, pushes, and cosign-signs the v0.19.0 images; the GitHub Release (tag0.19.0) follows and deploys the versioned docs; the sync workflow then rolls develop to0.20.0-alpha.0ahead of the Modular AirStack stack (#388–#396).🤖 Generated with Claude Code