Skip to content

Modular AirStack 6/9: vehicles, fleets, airstack.yaml, generic fleet spawner - #393

Open
andrewjong wants to merge 8 commits into
rfc/s5-macvo-flatten-splitfrom
rfc/s6-fleets
Open

Modular AirStack 6/9: vehicles, fleets, airstack.yaml, generic fleet spawner#393
andrewjong wants to merge 8 commits into
rfc/s5-macvo-flatten-splitfrom
rfc/s6-fleets

Conversation

@andrewjong

Copy link
Copy Markdown
Member

Stacked PR — merge the stack bottom-up with merge commits (squash-merging a lower PR orphans the commits every higher PR builds on). When the PR below merges, GitHub retargets this one to develop automatically. Each PR bumps VERSION (0.20.0-alpha.N), so check-version-increment passes at every level and each merge publishes images.

Stack 6/9 · base: rfc/s5-macvo-flatten-split

Heterogeneous AirStack scales 1–2 (RFC #380): fleets separate who exists from how each one flies.

What's inside

  • Fleet files (config/fleets/*.yaml): robot = vehicle (config/vehicles/) × stack × unit, plus hosts: placement of split-stack offboard halves onto ground hosts. airstack up --fleet <name> validates, derives NUM_ROBOTS, and selects the generic Isaac fleet spawner (fleet_spawn.py).
  • airstack fleet list|generate: per-robot compose generation for heterogeneous fleets (homogeneous keeps deploy.replicas); bridge router configs auto-generated per fleet; .airstack/generated/ mounted into robot containers.
  • Opt-in resolver (tools/fleet/resolve_fleet.py via FLEET_CONFIG_FILE): containers resolve their whole fleet entry; without a fleet, the replica-based path is byte-identical. airstack sync reconciles airstack.yaml.
  • Harness --fleet option; doctor --live handles heterogeneous fleets; airstack down sees generated compose services (fixes orphaned fleet containers); flight harness waits for the state-estimate watchdog before commanding takeoff.

Validation

  • Heterogeneous 3-stack fleet (full + lite + split) flight-ready in 100–111 s with a live bridged global_plan crossing the DDS router.
  • Fleet-resolver parity contract test (homogeneous ≡ legacy resolver); legacy NUM_ROBOTS multi-robot still green; single-robot fleets keep the historical single-drone prim names.

🤖 Generated with Claude Code

andrewjong and others added 8 commits August 22, 2026 04:02
… Isaac spawner

RFC #380 scales 1-2, all OPT-IN (no fleet declared = byte-identical behavior,
test-pinned):
- airstack.yaml: the one hand-edited entry point (release/fleet/sim/modules/
  external stack repos); airstack sync resolves it (module sync + stacks/
  .external fetch + fleet validation + effective_sources.yaml)
- config/vehicles/quad_default: pass-through vehicle (URDF generation future);
  sensors carry the real driver+sim pairing per RFC #380 §1; per-serial
  calibration in gitignored config/local/
- config/fleets/{sim_one_default,sim_three_mixed}: robot = vehicle × stack ×
  unit; sim_three_mixed exercises the SPLIT case (robot_3 on
  lite_offload_global with hosts: {offboard: gcs})
- tools/fleet/resolve_fleet.py (opt-in via FLEET_CONFIG_FILE; legacy resolver
  untouched and default; parity with it test-proven) +
  generate_fleet_compose.py (per-robot/per-ground-host services for
  heterogeneous fleets; homogeneity detection; split placement gives the
  ground host ENTRY=offboard; bridge hard-gate asserted)
- fleet_spawn.py: generic Isaac spawner driven by the fleet file (spawns,
  vehicle lidar toggle, scene); example_* scripts remain escape hatches
- airstack up --fleet / airstack fleet list|generate / harness --fleet;
  effective-config gains the resolved robot table; explicit env still wins
  with the override banner
- docs/development/fleets.md; configure-multi-robot rewritten fleet-first

Unit suite: 238 passed (+30, zero regressions).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ampaigns to fleet size

- ready.sh matched only compose replicas (-robot-); fleet-generated services
  (airstack-robot_N-1) were invisible, so heterogeneous bring-ups timed out at
  'robot containers running' with all containers actually up. Ground-host
  tenants (gcs-robot_N) are excluded — they never run MAVROS/PX4
- pytest_generate_tests now implements what the --fleet docstring promised:
  campaigns run at exactly the fleet's robot count instead of the
  --num-robots matrix (a 3-robot campaign against a 1-robot fleet failed on
  'no odometry' for robots the fleet never declared)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nerate bridge router configs

robot_3's onboard entry (split stack) loads the bridge-derived DDS-router
config at .airstack/generated/dds_router.<stack>.yaml — the dir was never
mounted and generation was a manual step. Both compose paths (robot-base and
the fleet-generated services) now mount it read-only, and fleet generate /
up --fleet materialize router configs for every split stack the fleet
references.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- discover fleet-generated robot services (same pattern fix as ready.sh);
  ground-host tenants excluded
- --live --stack <name> captures only robots whose AIRSTACK_STACK_DIR names
  that stack — one wiring.md describes one stack, not the fleet union
  (validated live against the running sim_three_mixed fleet: full_default
  diff shrank from robot_3's whole graph to a single honest environmental
  finding — the GCS action_relay client count differs between 1-robot and
  3-robot deployments; host-grouped wiring per RFC #380 is the refinement)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… names

fleet_spawn produced multi-style prim names for a 1-robot fleet, deviating
from the validated example_one configuration (/World/base_link +
PX4Multirotor) — the fleet parity flight stalled at takeoff. Single-robot
fleets are now byte-equivalent to the single script's drone config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…keoff

px4_ready proves EKF/MAVROS signals, but sending takeoff the instant it
passes races the drone_safety_monitor's watchdog after slow sim loads —
observed as 'Goal was rejected' and 'failed to arm' (three distinct race
signatures on fleet runs; manual takeoff on the settled stack accepted fine).
Wait up to 45s for state_estimate_timed_out=false; proceed with a warning if
unconfirmed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fleet-generated services were invisible to 'down' (only the base compose file
was loaded), leaving orphaned robot containers busy-looping at ~100% CPU each
(ddsrouters spin without a sim /clock) — which tanked the machine (load 70)
and caused every subsequent flight test to time out regardless of dispatch
path (legacy control reproduced it). down now includes the generated overlays
and passes --remove-orphans as a backstop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant