feat: optional TwelveLabs Pegasus clip selection - #7
Open
mohit-twelvelabs wants to merge 1 commit into
Open
Conversation
Rerank the Pexels stock-video candidates with the Pegasus video-understanding model so the most relevant B-roll is used first. Opt-in via TWELVELABS_API_KEY; when unset the original Pexels ordering is kept, so existing behaviour is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
This PR adds an optional content-selection step that uses TwelveLabs' Pegasus video-understanding model to pick the best B-roll for a Short.
What it adds
Backend/twelvelabs_select.py— a small provider module (rerank_clips) that scores each candidate Pexels clip 0-100 for how well it visually matches the video subject and script, then reorders them best-first.Shorts.DownloadVideos: right after the candidate URLs are gathered (and before download), the list is reranked. Pegasus reads the public Pexels URLs server-side, so there's no upload/index step.README.md,EnvironmentVariables.md,.env.example, andrequirements.txt(twelvelabs>=1.2.8).Backend/test_twelvelabs_select.py— stdlibunittest, four no-network unit tests plus one live smoke test gated onTWELVELABS_API_KEY.Why it helps
Today every clip Pexels returns for a search term is treated equally. Pegasus actually watches each candidate and ranks them by relevance to your subject/script, so the most on-topic footage ends up in the Short — better than keyword-match ordering alone.
Opt-in / non-breaking
Disabled unless
TWELVELABS_API_KEYis set. When the key is absent (or the SDK/network is unavailable),rerank_clipsis a graceful no-op and the original Pexels ordering is used, so existing behaviour is unchanged. It follows the same env-gated optional-provider pattern already used for the TTS engines.Tested
python -m unittest Backend/test_twelvelabs_select.py— 4 unit tests pass (live smoke skipped without a key).pegasus1.5.Note on process
The README asks to open an issue first for major changes. This is a small, self-contained, opt-in addition, so I went straight to a PR — happy to open a tracking issue or adjust anything if you'd prefer.
You can grab a free API key at https://twelvelabs.io — there's a generous free tier.