[Remyx Recommendation] Aurora: Unified Video Editing with a Tool-Using Agent

[github-actions]

Drafted by an autonomous discovery loop — Remyx ranks recent arXiv papers against this team's research interest and shipping history; Claude Code selects the candidate most directly implementable against this repo from the lookback window and drafts it.

Recommended paper: Aurora: Unified Video Editing with a Tool-Using Agent
Confidence: 🟢 high (Remyx relevance 0.98)
Research interest: FFMPerative
Implementation by: Claude Code as autonomous agent

---

Why this paper for this team

This paper is highly relevant to FFMPerative's 'Initial implementation of FFMPerative core and tools' by presenting Aurora, an agentic video editing framework. Aurora's VLM agent maps raw user requests to structured edit plans, resolving textual and visual underspecification before generation. This methodology provides a direct blueprint for how FFMPerative can refine its LLM agent to interpret natural language instructions and compose complex ffmpeg pipelines. The paper's focus on structured edit planning and reference-image selection is directly applicable to improving FFMPerative's tool selection and instruction-following capabilities for varied video processing primitives.

Why this candidate (selected from the lookback pool)

Aurora's implementable contribution is its VLM agent that maps raw NL requests into structured, underspecification-resolved edit plans — exactly what FFMPerative's interpretor.py + prompts.py + tool_mapping.py already do when turning a user request into an ffmpeg tool sequence. It ships as a prompting/planning refinement the repo can host as code it already calls, with no new model, trainer, or checkpoint required (the diffusion-transformer generation half is simply ignored).

Suggested experiment

Adapt Aurora's approach to map a specific natural language video edit instruction (e.g., "crop the video to 16:9 and add a fade-in") to a structured ffmpeg command sequence. Evaluate how well the FFMPerative agent, informed by Aurora's planning strategy, generates the correct ffmpeg pipeline compared to current methods.

---

What this PR delivers

Call site: ffmperative/interpretor.py::evaluate — check_edit_plan(expression, tools) is called right after parsing and before the evaluate_ast execution loop.

Delivers (from the paper):

• Implements Aurora's 'complete edit planning' step as a pre-execution check (ffmperative/edit_plan_check.py): the agent's generated tool sequence is parsed into structured PlanStep/EditPlanReport objects and each step is verified against the available tools before any ffmpeg primitive runs.
• Surfaces two concrete forms of underspecification up front — unknown tools and missing required arguments — using inspect.signature to compute each tool's required params and matching them against positional + keyword args supplied in the AST.
• Wires the check into interpretor.evaluate so every plan is validated on the normal execution path, printing the underspecified-step issues before the existing evaluate_ast loop executes anything.

Intentionally out of scope (not needed for this contribution):

• The diffusion/video-generation half of Aurora — the unified video diffusion transformer and reference-driven synthesis (requires a generative model FFMPerative does not host; the repo only composes ffmpeg primitives).
• Supervised training of the VLM agent for edit planning and reference-image selection, plus the preference pairs for robust tool use (requires a training pipeline and labeled data).
• The AgentEdit-Bench evaluation harness for measuring instruction-following under underspecification (requires benchmark data/infra).
• Active repair/rewrite of incomplete plans and resolution of visual underspecification — the check only flags textual/argument gaps; it warns but does not block or fix, and does not halt execution.

This delivers Aurora's core implementable insight — resolving underspecification by validating a structured edit plan before generation — adapted to FFMPerative's ffmpeg-primitive setting. The new edit_plan_check module parses the agent's tool sequence into structured steps and checks each for unknown tools and missing required arguments, and it is invoked from the pre-existing interpretor.evaluate entry point, so it runs on the normal execution path (not an orphan). Intentionally scoped out as unnecessary for that value are Aurora's generative diffusion-transformer half, the VLM agent training/preference pipeline, the AgentEdit-Bench evaluation, and active plan repair — the focused slice delivers plan-completeness verification over ffmpeg tools, which is the relevant result for this repo. One honest boundary: the check currently warns rather than blocking, so it reports gaps without preventing execution.

Test results

✅ All tests passed.

---

Opened by the Remyx Recommendation orchestrator.