VGGT Omega implementation#1116
Open
hkhanuja wants to merge 2 commits into
Open
Conversation
added 2 commits
June 23, 2026 02:18
kathirgounder
added a commit
to kathirgounder/gtsfm
that referenced
this pull request
Jun 27, 2026
Ports VGGT-Omega support from PR borglab#1116 (hkhanuja) — the VggtOmegaGeometryTransformer, the thirdparty/vggt-omega submodule (@39a0cb8), the gated-weights download flag, and the baseline omega optimizer/config — and runs omega geometry through OUR verified optimizer (ClusterVGGTWithFrontend, via its injected geometry_transformer) so it inherits global Fetzer focals, correspondence reuse, and the post-merge trackless-camera recovery (rather than Harneet's optimizer, which lacks those). Code wiring (2 small fixes): (1) cluster_vggt_with_frontend.py guards the two transformer.config dereferences (__init__ loader_kwargs + __repr__ cache key) so a transformer without a .config (omega) instantiates and caches cleanly; (2) vggt_omega_geometry_transformer.py adds a per-worker model singleton so the 1B weights load once per worker when the optimizer drives predict() with model=None. New config vggt_omega_sift_frontend_megaloc_phototourism_verified.yaml: the verified config with the geometry transformer swapped to omega and model_cache_key=false (forces omega self-load; null would silently load VANILLA VGGT). Controlled A/B vs the 0.6968/230-cam VGGT-verified run — only the geometry predictor changes. With omega's tighter poses, the trackless-recovery is the lever expected to finally recover 104/207. Requires the gated cc-by-nc-4.0 omega weights (HF facebook/VGGT-Omega) + 'git submodule update --init --recursive'. CUDA-only. Omega module imports are lazy (registry + Hydra), so non-omega runs are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
|
we should not need a new cluster_optimizer pipeline: gtsfm/cluster_optimizer/cluster_vggt_omega_with_frontend.py we are adding a new model, which is a new geometry_transformer. this should be supported in the existing cluster_optimizer pipeline. if this needs some changes to the API of the geometry transformer class, or the existing cluster optimizer pipeline, that can be done. let me know if that makes sense? If there is a logical reason why these cannot share the same cluster optimizer pipeline, please explain that here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the VGGT-Omega backend to the GTSfM pipeline.
Since the VGGT-Omega weights are protected under a license and cannot be distributed, the user needs to get access to the model and then download the weights using their
hf_tokenusingscripts/download_weights.sh --hf_token <your hf_token>Since the VGGT-Omega poses can be considered accurate, we add a factor to the prior camera poses. (can be experimented with)
We also set the camera type to Cal3DS2 to incorporate different focal lengths (fx and fy) and include distortion parameters.