Working with MGT python

ambiscape and the Musical Gestures Toolbox for Python (MGT) are sister toolboxes: ambiscape owns the samples, MGT owns the pixels. ambiscape analyses the sound of a place; MGT analyses motion in video, handles pose, and flattens and stitches every 360-camera format the lab uses. Together they cover an audio-visual field session end to end. This page documents the seams: the shared clock, device synchronisation, the one-directional bridge, and the per-camera workflow.

The MGT-side version of this page is Working with ambiscape in the MGT wiki.

Design rules (why the integration looks like this)

1. ambiscape never imports MGT. The core stays numpy/scipy/soundfile-light so that it runs in course CI and on thin machines; the single crossing point sits on MGT's side, behind an optional extra: pip install "musicalgestures[soundscape]".

2. One time contract. A recording's start time comes from a filename stamp (YYYYMMDD_HHMMSS / YYMMDD_HHMMSS, identical regexes in both toolboxes), a BWF bext chunk (ambiscape), or file mtime (last resort). Feature series carry their absolute start, so audio and motion join on the wall clock with a plain merge_asof.

   The contract binds what each toolbox writes, not only what it reads. io.export_segment names its output with the excerpt's wall clock, so a folder of exports re-opens as a session on the clock it was cut from; without that, mtime is the only fallback left and it dates a segment to whenever it was written.

3. Namespaced summaries. One analysis/summary.json per session: ambiscape's descriptors, vis_* keys from ambiscape vision --merge, and mot_* keys folded in by MGT's merge_into_summary(). No second dialect.

Multi-device sessions: measure with MGT, apply in ambiscape

Real sessions have per-device clock errors (a five-device belfry session: −5 s, +2 s, 0 s). MGT measures pairwise offsets by envelope cross-correlation:

from musicalgestures._sync import align_recordings
align_recordings("insta360_00.insv", "recorder.m4a")
# {'lag_s': 17.125, 'peak': 0.68}  # positive = second starts later; peak <0.3 = distrust

ambiscape applies them per take via calibration.json in the session folder:

{
  "clock_offset_s": 0.0,
  "clock_offsets_s": {"20260724_143753_insta039.wav": -5.0,
                      "20260724_143810_samsungaud.wav": 2.0}
}

Keys that match no session file trigger a warning, since a typo would otherwise silently misalign exactly the multi-device analysis this exists for. Field-side prevention beats both: clap once while all devices roll (see Field Recording Protocol).

A high correlation peak is not proof the alignment is right. Aligning against a GoPro's proxy .LRV instead of the full .360 reproduced the full file's offset exactly on three days out of four; on the fourth it was 60 s out, with a peak standing 50× above the correlation median — no threshold on peak would have caught it. What caught it was the signal: cut the window at the predicted position and look for the transient you aligned on. A real clap window shows 25–45 dB between the 20 ms after the onset and the 20 ms before it; the mis-aligned one showed −4 dB, i.e. room tone. Since the check costs one short decode, make it the gate and treat peak as a hint. The general lesson, if you take one: a method's confidence in itself is not evidence — verify against the thing being measured.

(The proxy is ~60× faster to read, which is why it is tempting. Use it to propose an offset, then confirm on the real file.)

The action begins before the sound — and that is the point of the pair

The strongest reason to run the two toolboxes together is not that pictures are nice to have beside a waveform. It is that a sound-producing action starts well before its sound: an intention becomes neural and then muscular activity, then motion in the arm and the object, and only at the end does the air move. By the time anything reaches a microphone, most of the action has happened.

That is measurable, and it has been measured. Giving MGT's motion series and ambiscape's audio energy the same onset rule, across 180 clips carrying both modalities of the same event:

from ambiscape import analysis
analysis.onset_lead(qom_from_mgt, audio_energy, dt=1/25)
# {'lead_s': 0.72, 'leads': 'first', ...}

Motion leads sound by a median 0.72 s, in 84 % of clips. The remaining sixth is real rather than error — an object already moving when struck, or an action out of frame, has no visible beginning.

So the lead is a property of the action, and neither modality carries it alone: it exists only in the pair. Two design notes follow, and one caution that replaces a third.

Keep motion where motion is computed — onset_lead takes series, not files, precisely so this stays a seam rather than one toolbox reimplementing the other. And read lead_s rather than the two onset times it returns beside it, which are the more fragile quantity.

The caution. This page used to advise passing rise=0.75 whenever an onset time was wanted. Do not, for a motion series: checked against onsets marked by eye from video frames, blind to the computed values, 0.10 and 0.25 land within a median 0.06 s of what a viewer calls the beginning, while 0.50 and 0.75 are late on every clip checked, by a median 0.46 s and 0.66 s. Motion has a genuinely still lead-in that a high fraction sits through until the movement is already large.

And do not treat 0.75 as established for audio either. The figure usually given for it — landing within 0.01 s where 0.25 lands 1.78 s early — compares the rule against another algorithmic onset, series_onset at 0.25 on a linear energy series, and not against anyone's ear. The two passes also used different representations, dB against linear energy, on which the same fraction lands tens of frames apart; the apparent agreement was substantially two mismatches cancelling. Nobody has yet marked an acoustic onset on this corpus by hand.

So one fraction applied to both modalities is wrong for at least one of them, and the fix is not simply to use two: only the motion fraction has a person behind it. Until the audio side is marked, prefer the symmetric default, quote what it costs, and do not publish a figure that turns on the audio fraction. Note also that the rule's high end is s.max(), one frame setting the crossing level for a whole series: varying only rise between 0.20 and 0.30 across 365 clips moves the onset more than half a second on 31 of them.

What ambiscape ingests directly

Session folders accept camera containers, not just WAVs: .mp4 .mov .m4a … .insv .lrv .glv .360 decode on ingest via ffmpeg. Two behaviours matter for 360 cameras:

• Best-stream selection: containers with several audio streams decode the best decodable one, where most channels wins. A GoPro MAX .360 therefore yields its 4-channel first-order AmbiX PCM track (verified B-format in the lab's spatial-audio study), not the stereo AAC sitting in front of it. Deep PCM sources cache at 24-bit.
• Camera caveats (from Riaz, Guo & Jensenius): Insta360 4-channel audio is not B-format, and ambiscape will still label 4-channel input ambix by count, so treat Insta360 spatial metrics as unreliable until that layout is established. Garmin VIRB 360 is AmbiX with an empty Z channel ("planar"): azimuth fine, elevation meaningless. Prefer in-camera originals over VIRB Edit exports (bugged FOA track). iPhone spatial captures carry an APAC track ffmpeg cannot decode, so ambiscape uses the stereo AAC unless the APAC is decoded to 4-channel PCM on Apple hardware first (afconvert, macOS 15+).

Video-side prerequisites (MGT)

Flattening/stitching lives in MGT, one call per camera family:

Camera	Call
Insta360 two-lens .insv pairs	stitch_dual_fisheye(front, back) (auto FOV calibration; a lab X4 measured 195°, not the nominal 190°)
GoPro MAX/MAX2 .360 (custom EAC)	flatten_gopro360(path) — probe-driven, accepts chunk-merged .mkv; MAX2 experimental
Ricoh Theta S legacy dual-fisheye	flatten_theta360(path) — explicit call; rotated lenses defeat plain v360
Garmin VIRB 360	nothing (already equirectangular); RAW 200° pairs → stitch_dual_fisheye

GoPro and Theta S flattening are validated on real footage (the lab's four-camera comparison data); Theta output matches the THETA app's export up to a yaw offset.

The bridge back: ambiscape features inside MGT

For joint analysis, MGT pulls an ambiscape session's 1 Hz features into its own container type, wall-clocked and ready to merge with motion series:

from musicalgestures._soundscape import soundscape_features, merge_into_summary

aud = soundscape_features("2026-07-24-belfort-session")   # MgFeatures, aud_level_db
merge_into_summary(motion_features, "…/analysis/summary.json")  # adds mot_* keys

This runs (or reuses) ambiscape.features.extract_session under the hood, which is the same cache ambiscape analyze builds, so nothing is computed twice.

A worked example

The Belfort (Bruges) session: five devices, 36.5 min, two rooms.

1. MGT's align_recordings fixed the device clocks; offsets went into calibration.json.
2. One ambiscape session over all five tracks: analyze → descriptors; escapement → the tower clock's 1.649 s beat, ±11 ms wander; carillon --events → 29 distinct strike notes across the quarter chimes; taxonomy → Schafer timeline + Schaeffer map.
3. MGT stitched the Insta360 pairs to equirectangular panoramas on the same clock, so every transcribed bell strike indexes straight into the 360 footage.

See also

• MGT wiki: Working with ambiscape—the same integration from the video side
• Field Recording Protocol—record so that alignment is trivial
• Recipes—ambiscape task walkthroughs
• Lab papers behind the camera facts: Guo, Riaz & Jensenius (SMC 2024, video formats); Riaz, Guo & Jensenius (spatial audio tracks)
• The papers' recordings are published as a dataset: 10.5281/zenodo.21611765—original files per camera with measured cross-device timing, useful for testing spatial-audio pipelines