[skip benchmarks]Vggt verified pipeline#1117
Draft
kathirgounder wants to merge 70 commits into
Draft
Conversation
…e retriangulation Gated behind a single SceneOptimizer flag `use_verified_pipeline` (default off, so existing configs are byte-for-byte unchanged). When enabled, SceneOptimizer.run: 1. Runs a global two-view verification pass over the full MegaLoc retrieval graph, reusing the per-cluster frontend chain (ClusterMVO._run_correspondence_generator -> _pad_keypoints_list -> _run_two_view_estimation, which already filters to TwoViewResult.valid()). This populates the per-pair caches, so subsequent per-cluster frontends cache-hit. 2. Partitions METIS on the verified subgraph (edges that survive verification) instead of the raw retrieval graph, and logs verified-vs-retrieval edge counts + how many cameras the largest-CC extraction would drop. 3. Builds global 2D tracks from the verified correspondences (get_2d_tracks) and, after the hierarchical merge, retriangulates them against the merged VGGT poses, runs BA (jointly refining structure + poses), and writes the result to results/merged_retriangulated/ with its own metrics -- alongside the unchanged results/merged/ for A/B comparison. Reuses existing machinery only (multi_view_retriangulate_from_2d_tracks, CppDsfTracksEstimator via get_2d_tracks, BundleAdjustmentOptions.run_simple_ba); new code is confined to scene_optimizer.py (one helper + the flag branch). Adds vggt_sift_frontend_megaloc_phototourism_verified.yaml (phototourism config + use_verified_pipeline: true) as the A/B treatment arm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…erified) - Fix instantiation crash: merging_options.ba_options.use_pose_prior (removed from BundleAdjustmentOptions) -> use_pose_prior_all_cameras (preserves intent: soft prior anchoring every merged camera to its own pose during merge BA). - Detector: SIFTDetectorDescriptor (OpenCV) -> ColmapSIFTDetectorDescriptor. - MegaLoc retriever: num_matched 15 -> 100, min_score 0.5 -> 0.15 (denser graph, matches the peak megaloc_sift_gp_single_pt config). - METIS: min_cameras_to_partition 12 -> 30, max_cameras 40 -> 70 (larger clusters, shallower tree). Applied identically to both arms so the A/B differs only by use_verified_pipeline. Full config audit (25 _target_ blocks vs class signatures) found use_pose_prior was the only invalid key; all others validate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…0/0.15 gate Brings the Brussels phototourism configs (baseline + verified) to the peak two-view frontend from the gp-glomap-parity lineage (commits da03cb8 "PoseLib verifier...", 5ed1955 "Brussels AUC@5 0.730->0.795"): - verifier: Ransac -> PoseLibVerifier (estimation_threshold_px=2). mapping-vggt's PoseLibVerifier is a functional superset of gp-glomap-parity's peak verifier (poselib.estimate_relative_pose: 5-point + LO-RANSAC + 2-view bundle), so no port needed. Dropped the Ransac-only use_intrinsics_in_verification key. (Peak uses PoseLib, not scipy -- scipy was removed in c9ed098.) - ColmapSIFT max_keypoints 5000 -> 8192. - inlier_support_processor 15/0.1 -> 30/0.15 (GLOMAP-matched edge-quality gate). - Declare poselib>=2.0 in pyproject (PoseLibVerifier imports it at module load). Note: not using the fetzer-only estimate_calibration_geometry variant (that param does not exist on mapping-vggt's PoseLibVerifier). Applied identically to both arms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…: true) Turns on mapping-vggt's existing scipy Fetzer focal-length refiner (view_graph_estimator/view_graph_calibration.py: scipy.optimize.least_squares on E=K2^T F K1 singular-value residuals, Cauchy loss). It recomputes F from the PoseLib-verified correspondences and refines per-camera focal lengths; VGGT poses are untouched, principal point/distortion fixed. Independent of the verifier and distinct from the gtsam Fetzer SelfCalibrationFactor (which is not on this branch). Left ba_options.use_calibration_prior = false (cluster + merge): the refined focal is used as the BA initialization; BA may still adjust it. Applied to both arms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d pipeline) In the verified pipeline, clusters now reuse the global verified two-view + SIFT tracks (filtered to each cluster's cameras) instead of re-running a per-cluster correspondence + two-view frontend, and build the cluster BA's 3D structure by triangulating those SIFT tracks against the VGGT poses (multi_view_retriangulate_from_2d_tracks) instead of lifting per-pixel VGGT depth. Fixes the trackless-camera problem: cluster-local frontends left many cameras with few/no tracks, and the VGGT-depth 3D init was multi-view-inconsistent so the pre-BA reprojection filter (14px) dropped most tracks (survival 0-76% per cluster). Global SIFT tracks give richer per-camera coverage; triangulation yields consistent structure that clears the filter. - ClusterContext gains an optional precomputed_global_frontend bundle (scattered futures: padded keypoints, verified two-view results, global 2D tracks). - SceneOptimizer scatters these once (broadcast) and threads them into every ClusterContext when use_verified_pipeline is on. - ClusterVGGTWithFrontend branches on the bundle: filters global tracks to the cluster's cameras (_filter_tracks_to_cameras), filters verified two-view to cluster edges for the scipy focal calibration (_filter_two_view_to_cameras), skips the per-cluster frontend, and builds BA input via _build_gtsfm_data_via_triangulation. Baseline arm unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…broadcast broadcast=True replicated the global keypoints + verified two-view results + ~80 MiB track list onto every worker, OOMing the node (KilledWorker -> driver dies at handle.metrics.result()). Scatter once (broadcast=False); workers fetch on demand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… edges calibrate_view_graph's no-edges early return did 'return list(initial_intrinsics)', yielding camera-index keys instead of the intrinsics dict. Downstream then indexed it as intrinsics -> Cal3Bundler(pose, <int>) TypeError, crashing any cluster with no valid F-edges. Surfaces under use_gt_intrinsics=false (weaker verification). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ated structure client.scatter of the global frontend crashed dask right after the cluster tree. A cluster's BA only uses within-cluster measurements, which the per-cluster frontend already produces (cache-hit from the global verification pass), so the global-track plumbing wasn't worth the scatter fragility. Revert to the per-cluster frontend for cluster tracks; keep the triangulated 3D init (the actual fix for the trackless-camera / pre-BA-reproj attrition) behind a new use_triangulated_structure flag (true in both phototourism configs). The global two-view verification stays (verified-graph partition + post-merge retriangulation, which takes the concrete global_tracks_2d again). - cluster_vggt_with_frontend: always per-cluster frontend; 3D init triangulate-vs-VGGT-depth via use_triangulated_structure; drop the _filter_* helpers. - scene_optimizer: remove scatter + PrecomputedGlobalFrontend; post-merge retri takes concrete tracks. - cluster_optimizer_base: drop the precomputed_global_frontend field + bundle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…& merge BAs Use VGGT only for poses, Fetzer only for focals, and stop every BA from re-optimizing the focal away from its Fetzer value. Global Fetzer (verified pipeline): - compute_global_view_graph_intrinsics() runs one Fetzer optimization over the full verified view graph (heuristic init, never VGGT focals), so camera i gets a single F_global[i] identical in every cluster. - ClusterContext.global_refined_intrinsics carries it; ClusterVGGTWithFrontend gains use_global_view_graph_calibration (flag/property/__repr__/selection, precedence global -> per-cluster -> raw VGGT); SceneOptimizer computes it once. - Also fixes the broken view-graph-calibration logger (get_logger()). Two-tier focal anchoring (both phototourism configs): - cluster BA: use_calibration_prior=true, focal_sigma=5px -- pin focals at the Fetzer value so per-cluster BA optimizes poses, not focals. Without this each cluster drifts the same camera's focal differently (focal/depth ambiguity), so parent vs child separator-camera focals diverge -> noisier Sim3 merge. - merge BA: use_calibration_prior=true, focal_sigma=10px -- looser, where poses are globally reconciled. Note: cluster ba_options is in __repr__ (cache key), so this invalidates the cluster cache -- a full recompute, which global Fetzer requires anyway. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…low fix)
The triangulated-structure path inflated each cluster to 11k-15k tracks (e.g.
C=14887, C_3_1=11090) -> 85-200s BAs on huge factor graphs -> worker OOM and
cascading dask FutureCancelledError ("lost dependencies" + nanny kill), with no
proven gain over Akshay's VGGT-depth baseline (219 cameras / AUC@3 0.675).
Flip use_triangulated_structure: true -> false in the verified config, reverting
per-cluster 3D init to lifting VGGT predicted depth (the known-good, low-memory
path). The focal-flow fix is unaffected: _build_gtsfm_data_from_vggt_depth applies
the global Fetzer focals identically (refined_intrinsics), and the cluster/merge
calibration priors anchor at that focal regardless of build path. Post-merge global
retriangulation (use_verified_pipeline) and global Fetzer are kept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-cluster frontend ran correspondence generation and two-view estimation as Dask tasks that themselves opened worker_client() and submitted per-image / per-pair sub-tasks, then gathered them (cluster_mvo._run_correspondence_generator /_run_two_view_estimation). On a single worker that nested gather holds the entire frontend working set resident at once -- every image's keypoints+descriptors, every pair's putative correspondences, every TwoViewResult -- as live futures. Over the global verification graph (14585 pairs / 236 images) that balloons to tens of GB and, when the worker tips, the nested gather cannot recover (no other worker holds the deps) -> FutureCancelledError "lost dependencies", fatal. It was always the same line. Replace the nested submission with inline execution, mirroring the existing synchronous ColmapCorrespondenceGenerator pattern: - DetDescCorrespondenceGenerator.generate_correspondences_inline(images, vg): detect once per image, match once per pair, plain loops over the cache-backed primitives (+ base-class stub). - two_view_estimator.create_two_view_results_inline(...): same per-pair kwargs as create_two_view_estimator_futures, but call run_2view directly in a loop. - cluster_mvo: both _run_* methods drop worker_client(); images now arrive as a normal Dask dependency (a list of image futures, auto-materialized by delayed) instead of being re-gathered inside a nested client. scene_optimizer global-verification call site updated to match. Computed-and-discarded item by item, so peak memory is bounded to one cluster's features rather than the whole graph. Method bodies don't change the optimizer repr, so existing cluster caches stay valid (no forced recompute). Frontend is now serial within a task (no per-pair Dask parallelism); cache-cold runs are slightly slower. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… measurements _build_gtsfm_data_from_vggt_depth kept a SIFT measurement only if VGGT per-pixel depth_confidence > 0 at the keypoint, so cameras whose keypoints fell in low-confidence VGGT regions (low-texture/transient content, or portrait images center-cropped to 518) got zero tracks at construction and were dropped at merge/retri. On Brussels this lost 5 connected cameras (idx 5/104/160/206/207) that have abundant verified edges and are richly tracked in GLOMAP. Decouple the gate: every verified SIFT measurement now enters the track; only conf>0 depth anchors the 3D-point init. The existing per-measurement 14px pre-BA reproj filter then prunes pose-inconsistent observations, so recovery self-selects on pose quality — cams with sound VGGT poses (5: 0.30deg, 104: 0.16deg vs GLOMAP) return with real BA-refined tracks, while bad-pose cams (206 ~10deg) correctly stay out. Track count is unchanged (same confident-anchor gate); only measurement count grows, so no factor-graph blow-up like the reverted global use_triangulated_structure path (ede5a9b). __repr__ gains an /allkpts cache token (when use_triangulated_structure=false) to force the cluster recompute. Also documents a reverted global-Fetzer median-fill of unrefined focals: the cameras Fetzer can't refine skew high-focal (true f/maxdim ~1.2) where the 1.2 heuristic is already accurate, so median-fill hurt 3 cams to help 1 (215). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er frontend) The verified pipeline ran the SIFT frontend twice: once globally (the two-view verification, which produces padded keypoints + v_corr_idxs_dict), then again per cluster. The second pass dominated wall-clock -- not from recomputing SIFT (it cache-hits) but from serially deserializing 3 disk caches (detector + matcher + two-view) over hundreds of pairs, redundantly across heavily-overlapping clusters (~60s/cluster x ~40 clusters). A cluster build only needs tracks_2d (it extracts v_corr, runs get_2d_tracks, and discards the two-view relative poses/F/configs -- VGGT supplies poses). The global v_corr + keypoints already live in the main process, where each cluster's create_computation_graph also runs. So when reuse_global_correspondences is set: subset the global v_corr to the cluster's edges and build tracks_2d EAGERLY in the main process, then pass it into the VGGT build and skip the per-cluster frontend entirely. Only the resulting per-cluster tracks_2d (~1 MiB) is embedded in the dask graph -- no scatter (which OOM'd before), no full-dict embedding. Per-edge v_corr is identical to the per-cluster frontend output (same edges, same two-view, same heuristic intrinsics), so this is a pure speedup; verify by flag-on-vs-off track-count + AUC parity. Plumbed via two new ClusterContext fields (mirroring global_refined_intrinsics); gated by the reuse_global_correspondences flag with a /gcorr __repr__ cache token so the cluster cache invalidates once and flag-on/off get distinct keys. Falls back to the per-cluster frontend when the globals are absent (non-verified runs). Enabled in the verified config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cameras with good VGGT poses but zero VGGT depth-confidence (e.g. Brussels 104/207) get no 3D track at per-cluster construction (the confident-anchor gate) and are dropped at the merge. Capture those filter-dropped good-pose cameras with their merged poses and inject them into the post-merge retriangulation, which is depth-confidence-independent: it re-triangulates their existing global 2D tracks against the good poses. Cameras that still fail to gain a >=3-view track are cleanly dropped (final filter retain decoupled from keep_all_cameras). Gated behind recover_trackless_cameras_in_retriangulation (off by default; on in the verified config). Adds a per-camera recovery diagnostic (recovered, retri_tracks, global tracks touching, max views). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ports VGGT-Omega support from PR borglab#1116 (hkhanuja) — the VggtOmegaGeometryTransformer, the thirdparty/vggt-omega submodule (@39a0cb8), the gated-weights download flag, and the baseline omega optimizer/config — and runs omega geometry through OUR verified optimizer (ClusterVGGTWithFrontend, via its injected geometry_transformer) so it inherits global Fetzer focals, correspondence reuse, and the post-merge trackless-camera recovery (rather than Harneet's optimizer, which lacks those). Code wiring (2 small fixes): (1) cluster_vggt_with_frontend.py guards the two transformer.config dereferences (__init__ loader_kwargs + __repr__ cache key) so a transformer without a .config (omega) instantiates and caches cleanly; (2) vggt_omega_geometry_transformer.py adds a per-worker model singleton so the 1B weights load once per worker when the optimizer drives predict() with model=None. New config vggt_omega_sift_frontend_megaloc_phototourism_verified.yaml: the verified config with the geometry transformer swapped to omega and model_cache_key=false (forces omega self-load; null would silently load VANILLA VGGT). Controlled A/B vs the 0.6968/230-cam VGGT-verified run — only the geometry predictor changes. With omega's tighter poses, the trackless-recovery is the lever expected to finally recover 104/207. Requires the gated cc-by-nc-4.0 omega weights (HF facebook/VGGT-Omega) + 'git submodule update --init --recursive'. CUDA-only. Omega module imports are lazy (registry + Hydra), so non-omega runs are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The omega run was feeding VGGT-Omega images preprocessed by the VGGT loader (518px/14-aligned, pad-only) because _load_vggt_inputs hardcoded load_image_batch_vggt_loader. Omega expects its own preprocessing (512px/16-patch-aligned, aspect-cropped) and the depth-lift indexes the model's depth map via the loader's original_coords, so the mismatch ran omega out-of-distribution and likely mis-indexed its depth (measured: pre-BA track survival 75% vs VGGT's 81% on the same cluster). Fix: add GeometryTransformer.load_image_batch (default = the VGGT loader); VggtOmegaGeometryTransformer overrides it to use load_image_batch_vggt_omega_loader (mode=balanced; omega's modes are balanced/max_size, not VGGT's crop/pad). _load_vggt_inputs now dispatches via transformer.load_image_batch, and both call sites (ClusterVGGT, ClusterVGGTWithFrontend) pass self.geometry_transformer. Provably a no-op for VGGT: VggtGeometryTransformer inherits the base, which is the exact prior load_image_batch_vggt_loader(loader, indices, mode=input_mode) call. Only an omega transformer changes the loader. NOTE: the prior (mis-preprocessed) omega per-cluster cache must be cleared before rerunning, since this fix does not change the optimizer repr/cache key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Decomposes the 16-commit branch into 6 self-contained feature units (verified pipeline, global Fetzer, peak frontend, correspondence reuse, trackless recovery, VGGT-Omega) mapping 1:1 to the clean branches we'll land into master, with per-unit commits/files/design notes, the config-flag reference, the Brussels A/B result (0.7191 AUC@3 / 231 cams), and known follow-ups (104/207 PnP). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per Harneet's PR review. In _build_gtsfm_data_from_vggt_depth, keypoints in the cropped-away margins (under crop / omega aspect-crop input modes) map outside the VGGT dense map. The old np.clip(..., 0, W_vggt-1) clamped them to the border pixel and read THAT pixel's depth/confidence — anchoring the track's 3D point from an unrelated edge location, and letting border garbage count toward the >=min_track_length confident-anchor gate. Now we bounds-check and 'continue', so only genuinely in-bounds depths anchor the point. Safe: all_measurements.append moves above the pixel computation, so the BA-constraint set is unchanged (every measurement still added); only the depth-anchor set drops border-clamped garbage. In-bounds keypoints are byte-identical (clip was a no-op when in-bounds). NOTE: build output changes for cropped images -> clear the cluster_optimizer cache before re-running. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tabase.db GTSFM's Dask SIFT frontend OOMs on large scenes (St Peter's: ~2.5k imgs, 123k edges -> worker killed at the global two-view gather). This reads the verified frontend from a COLMAP database.db built offline (GPU SIFT + vocab-tree matching), bypassing the Dask frontend. New: ColmapDBRetriever (gtsfm.retriever.ColmapDB) returns the db's geometrically-verified pairs; scripts/build_colmap_db.sh builds a db per scene; vggt_omega_..._colmapdb.yaml wires retriever+ColmapCorrespondenceGenerator into the verified pipeline (db path defaults to <dataset_dir>/database.db). Reuses the orphaned ColmapCorrespondenceGenerator (db keypoints + verified two_view_geometries, with resolution rescale). Surgical bypass in scene_optimizer.py: a new CorrespondenceGeneratorBase.produces_verified_correspondences flag (True only for ColmapCorrespondenceGenerator) gates a branch that reads keypoints + v_corr from the db in the MAIN PROCESS — no Dask two-view estimation, no client.gather of all per-edge results (the OOM). Lines 337-366 (verified graph -> tracks -> reuse -> Fetzer) run unchanged; non-colmapdb configs are byte-identical (flag defaults False). VERIFY-FIRST on St Peter's (reproj is the canary): (1) db keypoints must land in the loader's resolution frame (ColmapCorrespondenceGenerator scales by image.width/camera.width); (2) for pairs where COLMAP image_id order != GTSFM filename order, confirm two_view_geometry.inlier_matches columns aren't swapped. Bad reproj => one of these. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The verified-pipeline global two-view pass gathered all ~N full TwoViewResults
(three per-point BA reports + putative idxs each) into the client, which
OOM-killed the client process on RAM-limited single nodes (e.g. PACE 1xH200).
Only v_corr_idxs is consumed downstream on this path (relative poses come from
VGGT per cluster), so collapse to the {(i1,i2): ndarray} dict inside the delayed
graph via _extract_v_corr_idxs_dict and gather only that. run_2view still runs
identically on the worker, so TwoViewEstimatorCacher/DB writes and valid()
filtering are unchanged; downstream consumers get the same v_corr_idxs_dict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The lean-gather fix (prev commit) bounded the CLIENT, but the verified-pipeline
global pass still built the full {edge: TwoViewResult} dict on ONE worker before
reducing — ~48k results x three per-point BA reports each (10-48GB) OOM-killed the
worker on large scenes (St Peter's, 48k edges). Add create_v_corr_idxs_inline +
ClusterMVO._run_two_view_v_corr_idxs: run run_2view per pair but keep only each
valid edge's v_corr_idxs and drop the heavy TwoViewResult immediately, so the
worker holds ~one result at a time instead of all N. run_2view still executes
identically (TwoViewEstimatorCacher/DB writes + valid() filtering unchanged);
_run_two_view_estimation and the per-cluster path are untouched (backward-compat).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kathirgounder
changed the title
…k worker) The verified-pipeline global frontend ran as one monolithic delayed task on one worker, with the main process blocking on a single client.gather for the entire multi-hour run. On large scenes (St Peter's, ~2500 imgs / 48k edges) the worker was repeatedly dropped mid-run with 'lost dependencies' + register-client comm churn. Confirmed NOT memory: node has 2TB RAM, frontend footprint ~15-20GB, and it died identically at 90GB and 128GB worker limits with no dask memory/GIL/segfault warning in the log. It is a scheduler<->worker comm/coordination failure over the long run. Fix: mirror the COLMAP-DB branch and run the frontend inline in the main process -- gather the downsampled images once, then call _run_correspondence_generator + _pad_keypoints_list + _run_two_view_v_corr_idxs directly (they are plain, no worker_client). No worker runs the frontend, so there is nothing for a comm hiccup to drop. Streaming v_corr reduction + TwoViewEstimatorCacher/DB writes unchanged; memory-trivial on a 2TB node; still serial (cache makes reruns fast). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… timeouts)
St Peter's (~2500 imgs) died in the image-loading phase with CommClosedError on the
client<->scheduler comm: get_image_futures submitted 2500 images in a tight
client.submit loop, which starves the client event loop between submits so it can't
drain the scheduler's key-in-memory replies -> the batched comm backs up and closes
mid-submission (seen at loader-get-image-1401), then the downstream gather hangs on
never-submitted futures.
- loader_base.get_image_futures: bulk-submit via client.map (one update-graph message)
instead of a per-index submit loop. Preserves keys (loader-get-image-{idx}) + worker
pinning. Verified client.map accepts a list key on distributed 2025.9.1.
- runner: raise distributed.comm.timeouts.connect/tcp 30s -> 300s so transient event-loop
stalls (bulk loads, the in-process frontend) don't tear down comms mid-run.
Complements the inline-frontend fix (1e1cc46): that removed the worker-side coordination,
this removes the client-side image-load storm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inline global frontend goes dark for tens of minutes with no signal. Add periodic progress logs (count, elapsed, rate, ETA) to the three serial loops so the multi-hour run is observable: - det_desc_correspondence_generator.generate_correspondences_inline: detection (every 250 imgs) + matching (every 5000 pairs). Adds a module logger (had none). - two_view_estimator.create_v_corr_idxs_inline: two-view (every 2000 pairs, + valid count). Logging only; no behavior change. Restart is cheap — the per-item cachers replay completed detection/matching/two-view instantly, so you get live progress from where it left off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Loc top-K COLMAP's vocab-tree matching returns a dense verified graph (~140k pairs on St Peter's -> 308k tracks -> a huge Metis cluster tree). ColmapDBMegaLocRetriever keeps only pairs that are BOTH COLMAP-verified AND in MegaLoc's per-image top-K (strict intersection), sparsifying the view graph while still reading correspondences from the db for the surviving pairs. Composes the existing ColmapDBRetriever + SimilarityRetriever (same pattern as JointSimilaritySequentialRetriever); no new pair-selection logic. - retriever/colmap_db_megaloc_retriever.py: the hybrid class (set intersection of the two retrievers' pair sets; both return i<j over the same index space). Guards the no-MegaLoc case by returning all COLMAP pairs. - retriever/__init__.py: register the ColmapDBMegaLoc short alias. - configs/..._colmapdb_megaloc.yaml: copy of the colmapdb config, re-enabling MegaLoc (GlobalDescriptorCacher) + ColmapDBMegaLoc retriever (num_matched=20, min_score=0.40). The dense colmapdb config is kept as the baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The global view-graph calibration called scipy least_squares with a vectorized residual but NO jac/jac_sparsity, so scipy built a DENSE numerical Jacobian: it perturbed every one of the N camera focals per iteration (N x batched-SVD-over-all-edges) plus a dense (2*n_edges x N) solve. Fine at brussels scale (~234 cams / ~2k edges) but it hangs at St Peter's scale (2504 cams / 117k edges). Each edge's two residuals depend on ONLY that edge's two focals (idx1, idx2), so the Jacobian is 99.9% sparse. Build that pattern from the already-precomputed idx1/idx2 and pass jac_sparsity= to least_squares, so scipy estimates the numerical Jacobian with graph-colored group perturbations + a sparse solve -> seconds instead of hours, at any camera/edge count. Pattern verified correct on a tiny example; residual math unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tersection collapse) Strict intersection of MegaLoc top-K with COLMAP-verified collapsed to 6294 pairs on St Peter's (MegaLoc + COLMAP rank neighbors differently -> ~13% overlap), fragmenting the graph (largest_cc 2504 -> 1945, dozens of tiny components) and giving Metis a degenerate tree. Switch semantics: rank each image's COLMAP-verified neighbors by MegaLoc and keep the top num_matched (restricted per-image top-K, reusing pairs_from_score_matrix with a COLMAP-only candidate mask). Guarantees ~K verified neighbors/image -> connected, near-full coverage, still MegaLoc-sparsified. Config min_score 0.40 -> 0.0 (rank-only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on pickle) The per-cluster dask graph captures the cluster_optimizer, which holds the ColmapCorrespondenceGenerator, which holds a pycolmap.Database handle — and pycolmap._core.Database can't be pickled, so client.compute() on each cluster crashed with 'cannot pickle pycolmap._core.Database' -> 'Could not serialize _HLGExprSequence' (scene_optimizer.py:252). This blocks every colmapdb run at cluster scheduling (reuse_global_correspondences means the worker never even uses the generator — it's just captured in the graph). Add __getstate__ that nulls the db handle + the large re-derivable keypoints cache, plus _open_db/_ensure_db so the db is re-opened lazily only if a worker actually calls the generator. Verified the pattern makes the object picklable; main-process behavior unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…line fallback) The verified-pipeline global two-view frontend ran serially in the main process (commit 1e1cc46), which was robust but did not use the worker pool. Replace it with a worker-count dispatch: - num_workers == 1 -> the existing in-process inline path (no worker to lose, zero scheduler<->worker comm-failure surface). - num_workers > 1 -> a new parallel path: the Dask correspondence generator (per-image detection + per-pair matching) plus create_v_corr_idxs_futures. create_v_corr_idxs_futures chunks the pairs (~4 chunks/worker) and runs the existing create_v_corr_idxs_inline as each chunk's body on the worker pool, returning only lean v_corr_idxs. Shared read-only inputs (estimator, keypoints, priors, mesh, per-view data) are scattered ONCE, each as a single blob (a bare scatter of a list/dict would explode it into per-element futures). Because the work is many small chunks rather than one monolithic task, a dropped worker only recomputes its own chunk instead of failing the whole multi-hour frontend. broadcast defaults to True: replicating the scattered blobs to every worker is both faster (no per-task fetch) and more robust than a single holder, since the raw scattered data is non-recomputable and would be permanently lost if its sole holder died -- the exact failure mode this frontend exists to avoid. Cheap on a big-RAM single node for sparse (<40k-edge) 1DSfM view graphs. Adds tests/test_create_v_corr_idxs_futures.py: parallel output equals the serial inline reference across default/tiny/single-chunk sizings, and the valid() filter is honored. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…at sigma=1.0)
The gid-anchored merges were still producing the same mis-seated floaters
because the point-correspondence noise sigma passed to TrajectoryAlignerSim3
was a fixed 1.0 -- about 25% of a typical cluster's entire diameter -- while
the shared-camera pose unaries run at ~1e-2/sqrt(N). Information ratio ~1e4:1:
the solver effectively ignored every point anchor and seated children on the
same concentrated pose hinges as before (lever-arm tilt unchanged, no matter
how many global-track-id correspondences we supplied). This was flagged by the
metrics-audit verification ("the 150 points barely influence the seat") and
not acted on when the gid anchoring shipped.
Set the sigma scene-scale aware: 2% of the parent camera-spread radius
(median distance to centroid), floored at 1e-4. Spread gid anchors now carry
comparable-to-dominant weight vs the pose hinge, pinning the far end of large
child blocks. Guard, scale band, and escape clause unchanged as backstops.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…loor Diagnosis of the sigma-weighted run (synthetic ground-truth sweeps + collision measurement on real exports): - With CORRECT pairs the aligner is stable at any sigma (worst 1.07 solved scale incl. clumped/low-count/outlier-spiked configs) -> the observed scale-81/289 divergences of 150-correspondence children require pair sets that are COHERENTLY wrong, not noisy (bin-collision rate measured at only 0.3-0.6% of measurements). - The 1.6e22px root detonation traced to a 66-camera child holding ONE track admitted through the overlap-escape clause (pose anchors can place a block; they cannot vouch for a hollow one). Changes: 1. _prefilter_point_pairs: 3-point RANSAC Umeyama over the pairs BEFORE the joint solve — removes outlier matches AND measures the similarity the pair set actually implies. If the pairs imply a child->parent scale outside [0.25, 4], the child is refused pre-solve with evidence (pair_scale, inlier_ratio) instead of detonating the merge and getting caught one level too late by the post-solve band. 2. point3_sigma 0.02 -> 0.10 x parent camera-spread radius (synthetic-validated stable; still ~100x the information of the old fixed 1.0). 3. Overlap escape now requires child.number_tracks() >= 50: structure floor. Post-solve scale band, 0-track drop, corr floor unchanged as backstops. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… retri off Scoreboard honesty: the most complete ToL reconstruction (R3: 471 cams, full outer wall + corner + bridge; GT-eval Nc=424, median 1.71m) came from the LEGACY merge with triangulated structure. The gid-anchoring/guard/sigma iterations since (d87e756..7af6437) consistently traded structure for accuracy (Nc collapsed to ~217) without eliminating the floaters. Completeness first: return to the best-known state, remove moving parts, then clean the OUTPUT instead of refusing merges. - SceneOptimizer.enable_gid_merge_anchoring (default False): gates building the gid index; without sidecars every merge runs the legacy shared-camera path (no corr-floor guard, no prefilter) and point sigma reverts to the legacy fixed 1.0. The gid machinery stays in-tree as an opt-in experiment, to be debugged via local replay before it earns another PACE run. - SceneOptimizer.run_post_merge_retriangulation (default True): gates the post-merge retri+BA stage; OFF in the ToL config while re-establishing the structure-complete baseline. - Kept active in both modes (tiny safety nets, only deltas vs R3): 0-track child drop and the post-solve Sim3 scale band (a no-op at sigma=1.0). - New test: sidecar-less merge keeps a large low-correspondence child (R3 semantics), alongside the 6 existing gid-path tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…graph, calibrations Enables desk-side experimentation on a downloaded results folder (merge strategies, BA pin/GNC matrices, prior/focal studies, cluster-Sim3-averaging prototypes) without burning cluster runs. All dumps are small and guarded: - global_tracks_2d.npz (~25MB): flattened (cam, u, v, track_id) for every global 2D track measurement — the global track IDENTITY that per-node COLMAP exports cannot express (needed for gid-correspondence debugging and cross-cluster constraint prototypes). Requires the numpy import scene_optimizer was missing. - cluster_tree.pkl: the exact partition tree (which node merges into which). - verified_graph.npz: verified view-graph edges + per-edge inlier counts (connectivity / edge-addition studies). - calibrations.json: per-camera loader-initial vs global-Fetzer focals (focal & calibration-prior experiments). Together with the per-node COLMAP text exports (vggt_pre_ba/vggt/merged_pre_ba/ merged), a full results folder is now a complete offline replay kit: every merge and every BA in the tree can be re-run locally under alternative strategies and scored against ground truth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t gid anchoring, boundary recovery
Offline replay on the instrumented Tower-of-London drop (exp5) validated all three
end-to-end: strays 54->30, aligned inliers 230->289.
1. Majority-vote track identity (cluster_merging._track_gid): a track's global id is
now the argmax over votes from ALL of its measurements, not the first index hit.
First-hit was dishonest — one pixel-bin collision on the first indexed measurement
assigned the whole track a foreign gid, minting phantom parent<->child Sim3
correspondences (offline: zero-overlap children arrived with "150 matches").
Voting makes rare collisions harmless and anchor counts honest, which is what the
MERGE_GUARD thresholds assume.
2. Boundary recovery stage (scene_optimizer._run_boundary_recovery, new flag
enable_boundary_recovery, default False): post-root-merge, on a worker (single
dask task, mirrors the retri stage):
a. read the merged scene's own 3D as {gid: xyz} via the majority-vote identity
(union gid sidecar riding the root scene);
b. DLT-triangulate every boundary global track with >=3 posed views and <3px
mean reprojection (faithful port of exp5 triangulate());
c. RANSAC-DLT resect unposed cameras (min 10 inliers @4px; exp5 resect()) against
the cluster-3D UNION fresh-3D — the union is essential (offline: an island
camera had 508 anchors in cluster-3D vs 95 fresh-only); cluster 3D wins on
gid collision; iterate b<->c (max 3) so new poses enable new triangulations;
d. add recovered cameras (COLMAP w2c -> gtsam cam-to-world) + boundary tracks
restricted to posed cameras, one BA via merging ba_options, post-BA 3px filter.
Intrinsics for unposed cameras: global-Fetzer focals, loader-initial fallback
(same precedence as the offline replay). Output: merged_boundary_recovered/ +
*_boundary_recovered metrics.
3. Re-enable gid merge anchoring in the verified config (safe now that identities are
majority-voted); enable_boundary_recovery: true; retri stays off. All existing
guards / RANSAC prefilter / sigma logic unchanged.
Tests: test_gid_merge.py extended with a majority-vote collision test and DLT
triangulation/resection unit tests (exact recovery on noise-free geometry); 10/10 pass.
test_create_v_corr_idxs_futures.py: 3/4 pass with 1 pre-existing flake (fails
identically on the clean tree, varying test identity; path untouched here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eak-structure A/B, tri-structure kept Threshold study: coverage is num_matched-limited (reference median degree 90); min_score irrelevant below K~160. 0.10/200 retrieves most of the reference's 306 C_9<->core bridge edges (we had 63) -> bridge seats on real evidence. Legacy merge (gid off) = peak-structure semantics for a clean A/B; majority-vote _track_gid stays in code. ~115k pairs expected (~4x two-view work, parallel frontend handles it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tracks whose max pairwise triangulation angle is < min_triangulation_angle_deg (config: 1.5, COLMAP's cutoff) are depth-unconstrained: they pass the 3px reproj filter with depth error hidden along the viewing ray and scatter as fuzz/spray. Validated on the ToL 120/0.15 drop: drops 5,806/35,305 tracks (16.4%), output matches the offline prototype the render was judged on exactly (29,499 kept). Output-side only: merged/ still exported unfiltered, poses/merges untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ded 7/8 clusters) Grading every GT-gradeable cluster of the ToL 120/0.15 drop showed per-cluster BA made poses WORSE vs GT in 7 of 8 clusters (median 2.90->4.34m; C_4 0.86->3.23m, C_9_1 11.4->19.7m). Mechanism: pre-BA structure reprojects at ~4px from intrinsics-model error (Fetzer K vs VGGT geometry); with focals pinned (5px prior) the BA resolves that inconsistency by moving the free poses — trading meters of world accuracy for a 0.3px reprojection. With BA off, every cluster ships bit-identical global-Fetzer intrinsics per camera (the strongest form of the cross-cluster focal coherence the prior was protecting), and structure polishing happens only in the pose-pinned merge BAs where it is safe. The flag rides the cluster cache key via repr (clusterba=...). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l prior 10->2 Gold audit vs 1DSfM GT (all 592 cams, exact per-image scales from the original images): loader/EXIF focals are 1.91% median error with +0.06% bias, while the scipy global-Fetzer values the pipeline froze are 5.26% with a -4.0% systematic bias (74% of cams >3% wrong; Fetzer degraded the very EXIF it started from). Mechanism: 56% of ToL's verified edges are Fetzer-degenerate (planar or focal-unstable per PoseLib H/F + interior-minimum gates) and the ungated joint solve feeds them all in. The 1DSfM reference pipeline itself calibrated from EXIF and excluded no-EXIF cameras. calibration_source rides the cluster cache key via repr (src=...). Merge focal prior tightened 10->2px: merge BAs were measured forking duplicated cameras' focals 9px median / 386px max across sibling branches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ge focal sigma back to 10 Live A/B verdict: the BA-off arm regressed (lost bridge, new floaters, flipped cameras). The cluster-BA census was survivor-biased — BA degrades the accuracy of cameras that survive, but BA+3px filtering IS the pipeline's early culling stage; without it flipped/garbage VGGT cameras ride to the final model and the merge BAs exclude rather than fix them. Net config vs the golden 404-cam run is now a single variable: calibration_source=exif (focals 5.26% biased -> 1.91% unbiased). BA-off stays available behind the flag for research with a replacement culling stage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The residual "hard gimbal lock" was ArcRotate fighting a mismatched up: ./viz used a Z-up upVector + per-axis Y-flip hack, so orbit's up axis didn't match the scene's real vertical. Port pipeline-viz's proven fix — align the data's true up to Babylon +Y once per load, then bake that rotation into every position: - Parse points/cameras in the RAW data frame (drop the Y-flip). Cameras now carry dataCenter (= -R^T t) + R (cam_from_world, plain 3x3 array). - _computeUpAlignment: scene up from mean camera image-up (-row1(R)); PCA least-variance fallback; flip so landmarks sit above cameras. Build an orthonormal basis mapping that up to +Y. - loadScene bakes the rotation into a NEW transformed points array (never mutates the cached raw array -> no double-transform on cache hit) and into camera centers. - Frustums: corner = apex + sceneRotation . (R^T . localOffset), matching pipeline-viz. - Framing: median center + 90th-pct range x2.5, camera on the photographer side at 30 deg elevation; snapshot as savedView for the R key. - Camera: default Y-up (no upVector), zoomToMouseLocation + useNaturalPinchZoom for exploration, and cleared/re-added inputs (Babylon's default multi-touch twist misbehaves on macOS trackpads). R now restores the saved framing. Splat rendering, scene sidebar, caching, and app.py contract untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Roman Forum frontend died unpacking knnMatch(k=2) results: OpenCV returns 1-tuples when the train image has <2 valid descriptors (blank/blurred internet photos). Skip queries that can't participate in a ratio test. ToL never had an image degenerate enough to trip this. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… tracks directly New scripts/convert_wilsonkl_frontend.py converts a wilsonkl dataset release (EGs.txt / coords.txt / tracks.txt) into an npz of loader-frame keypoints and per-edge verified correspondences (Roman Forum: 70,187 edges / 13.2M corr in 18s). SceneOptimizer(precomputed_frontend_path=...) loads it in place of the global frontend — retrieval/SIFT/matching/two-view skipped entirely. This is the standard 1DSfM protocol: every published method in the benchmark table consumed these released two-view inputs, so rows produced this way share identical frontend inputs with the baselines (ToL remains our end-to-end own-frontend result). Known caveat: coords are Bundler pixel-frame; images with EXIF-rotation may mismatch the loader frame and lose their edges at the triangulation filters (soft failure, minority of images). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bundler coords are raw-pixel-frame; the loader EXIF-transposes. Converter now stores expected loader dims per image; at load, images whose actual loader dims (from the EXIF-intrinsics principal points) disagree are dropped with a logged count — converting silent transposed-keypoint poison into measured coverage loss. Also hard-fails on converter/run max_resolution mismatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…anar/focal-sanity gates Ports the load-bearing elements of the SelfCalibrationFactor PR into the view-graph calibration: robust LO-RANSAC F via PoseLib (replacing 8-point on inliers), the H/F planar-config gate, the interior-minimum focal-sanity gate, and edge filtering scored at applied focals. Solver uses gtsam.SelfCalibrationFactor when the (custom) build provides it and falls back to the existing sparse scipy Cauchy solve otherwise. Gold-graded on ToL (592 GT cams, exact per-image scales): gates ON 4.31% median focal err, bias -2.64% gates OFF 14.32% median, bias -13.50% (56% of edges are Fetzer-degenerate) gate counts: 8324 kept / 2598 planar / 7949 focal-sanity of 18871. use_robust_gates=True by default; calibration_source=fetzer users (e.g. IMC phototourism, stripped EXIF) get the gated recipe automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t (e.g. 1DSfM EGs.txt) Hybrid protocol between own-frontend and precomputed-frontend: retrieval is skipped and SIFT (8192 kp) + matching + two-view run on exactly the benchmark's released pairs — full own-frontend quality at ~60% of the matching cost, with zero wasted compute on pairs that would die in verification (survival on speculative retrieval pairs was ~36%). Roman Forum: 70,187 EGs pairs vs 112,897 retrieval pairs, with SIFT + a chunk of matcher cache already hot from prior attempts. Motivated by the precomputed-frontend coverage failure: the benchmark's released keypoints (~1.3k/image, tracked features only) starve small clusters (53 MERGE_GUARD drops, Nc 267/1134) while its PAIR LIST is gold — so keep their pairs, recompute our correspondences. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bscription) BFMatcher parallelizes over cv2's global thread pool (default: all cores), so every concurrent apply_matcher task spawned a full-node pool — with 5-16 dask workers, 5-16x oversubscription explains both the slow matching stage and why adding workers made it WORSE. Same bug family as pycolmap SIFT num_threads=-1 (818f7b4). Set in match() since __init__ doesn't re-run in worker processes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cher Drop-in alternative to the OpenCV TwoWayMatcher with identical semantics — validated on real production SIFT descriptors (Brussels cache): match-set Jaccard 1.0000 across all tested pairs. One distance GEMM + topk per direction; ratio test applied in BOTH directions before the mutual intersection, exactly mirroring the OpenCV path (the unidirectional version disagreed at Jaccard 0.35 on real data — caught by the cache A/B). 2.2x faster than OpenCV even on 1 CPU thread (854 vs 1873 ms/pair at ~10k kps); ~2ms/pair on datacenter GPUs, which are idle during the matching stage. CPU fallback pins torch to 1 thread (dask workers are the parallelism). Opt-in via config/CLI: ...matcher.matcher_obj._target_=gtsfm.frontend.matcher.torch_twoway_matcher.TorchTwoWayMatcher Note: new matcher repr => matcher/two-view caches recompute (cheap at GPU speed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… verified config
Roman Forum audit: the [0.25, 4] band executed 18 lawful children (~1,200
cameras including a 258-cam subtree at scale 4.06; others at 4.7-12.5x and
0.07-0.25x). The band's premise ('metric clusters, scale ~ 1') does not hold:
VGGT normalizes scene scale per batch, so clusters with heterogeneous spatial
extents legitimately need large scale corrections at the seat. The detonations
the band was built for were 1e8+ — [0.02, 50] still rejects those by orders of
magnitude while admitting lawful sprawling-scene seats. Both guard sites
(pre-solve pair prefilter + post-solve) now read MergingOptions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The band's premise (metric clusters, solved scale ~ 1) is false: VGGT normalizes scene scale per batch, so lawful seats on heterogeneous scenes routinely solve at 4-12x (Roman Forum: 18 lawful children executed, ~1,200 cams). Genuinely diverged seats remain covered by the structureless/0-track guards, the correspondence floor, the pose-pinned robust merge BA, and the post-merge reprojection filters; NaN scales still drop (fail the comparison). Set a finite band in MergingOptions to restore legacy behavior. Pre-team-grind audit: 27/28 targeted tests pass (1 = known pre-existing flake in combined runs only, passes in isolation); all edited modules compile; torch matcher remains opt-in; vanilla IMC config inherits only (a) gated Fetzer and (b) band removal — both validated against 1DSfM gold GT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
For multi-node parallel experiments sharing one cache: - All six cachers now resolve their root via cache_utils.get_cache_root(), honoring the GTSFM_CACHE_ROOT env var (default: <repo>/cache, unchanged). Dask workers inherit the launcher's environment, so exporting the var at launch covers the whole worker fleet. - write_to_bz2_file is now atomic (unique tmp + os.replace): concurrent same-key writers and killed workers can no longer leave torn .pbz2 entries (observed in production caches). Readers already self-heal on corruption (delete + recompute); now they never see partial files in the first place. Verified: default root unchanged; override reaches cacher module constants in fresh processes; atomic round-trip; corrupted-entry self-heal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…+ double/Vector1 storage The closed-form focal factor appears as SelfCalibrationFactor in current gtsam-develop and FetzerFactor in pre-rename builds; older builds store the focal variable as Vector1 (optimize() throws GenericValue<double>-vs-Matrix). _solve_focals now resolves the class by either name and retries with Vector1 storage on failure, so any gtsam vintage with the factor gets the gtsam path automatically (scipy fallback unchanged otherwise). Validated end-to-end in the gtsfm-v1 env (FetzerFactor + Vector1): identical solve to the manual PR-parameter run on 6,035 gated British Museum edges. On those edges vs IMC GT intrinsics: gtsam 3.28% median focal err / 52% >3% vs scipy 4.06% / 62% — modest but consistent solver win. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Config-reachable sigma for the PriorFactorPose3 added by use_pose_prior_*, anchored at the initialization poses. Near-zero (1e-3) freezes poses so the cluster BA refines only structure+calibration — the self-calibration recipe validated offline on British Museum (Fetzer 8% focal error -> 1% after pose-frozen focal-free cluster BA; full-tail replay AUC@10 0.429 -> 0.677). Default (0.1) matches the previous ctor default: zero behavior change. NOTE: BundleAdjustmentOptions repr feeds the cluster cache key, so pulling this commit invalidates existing cluster caches (one-time VGGT recompute). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The calibration/pose priors that protect the incremental merges throttle the full-scene retri BA. retri_free_ba drops them (GNC on) for a fully-free final solve, GLOMAP-style; retri_iterations alternates retriangulation with the BA. Offline BM replay: full AUC@3 0.232->0.546 (baseline clusters) and 0.504->0.574 (selfcal clusters) on identical merged inputs; @10 0.477->0.794 / 0.770->0.818. Defaults (false, 1) preserve existing behavior exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Defaults (false, 1) = current behavior; keys present so campaign runs can override without '+'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GNC ablation on BM: +0.01 AUC for ~2x solve time — the win is from removing the priors (0.770->0.806 @10 constructed with the config's plain GM kernel). GNC stays reachable via merging_options.ba_options.use_gnc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Frustums: smaller (0.01 -> 0.006 of scene extent) and near-black (Color3(0.1) instead of orange), 1DSfM/Snavely wireframe look. Camera-center dots also darkened + shrunk to match. - New H hotkey toggles a "clean view" that hides BOTH overlays (the top-left stats card #sceneStats and the bottom control bar #hud) for an unobstructed screenshot; press H again to restore. Wired through _applyStatsVisibility / _applyHudMode so it survives scene reloads and stats/hud state changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The build overrides VGGT/Omega's predicted intrinsics with the view-graph focals, making them unrecoverable from any export. Capture them (rescaled to original resolution) before the override and write vggt_predicted_intrinsics.json next to the vggt/ exports. No cache-key change; cached clusters simply skip the sidecar. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion Bring the pipeline-viz branch's modern viewer in as a second, standalone viewer alongside ./viz (which is untouched). The branch version only handled pipeline traces (manifest.json + stage_*) and was missing its templates/index.html, so it couldn't run at all. Adapt it to render STATIC reconstructions: - server.py: find_runs() discovers static reconstructions (any dir with points3D.txt, excluding trace stage_* dirs) in addition to pipeline traces; the manifest endpoint synthesizes a single-stage manifest for static recons; default --base is now `results` (same data ./viz uses). Port 5174. - templates/index.html: NEW — the DOM shell viewer.js expects (stats card, help card, control bar) styled to the existing Snavely-inspired style.css. Trace-only controls (mode/play/timeline/stage) carry data-role="trace-only". - viewer.js: build stage URLs that tolerate an empty subdir (static = dir root); hide the trace-only controls when a run has a single stage; add an H hotkey to hide all overlays for clean money shots (matches ./viz). Server path verified end-to-end locally (discovery -> synthesized manifest -> data serving -> index render, all 200s). Run: ./pipeline-viz --base <results>. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Just creating draft pull request into the mapping-vggt branch so people can see experimental changes:
PR #1117 — VGGT Verified Pipeline
16 commits · +1913 / −80 · 23 files · targets
mapping-vggtThis branch turns the per-cluster VGGT reconstruction into a verified-view-graph pipeline and stacks
four accuracy/robustness features on top of it, ending with a VGGT-Omega geometry arm. It's large
because it was developed as one experimental thread; this doc decomposes it into 6 self-contained
feature units that map 1:1 to the clean branches we'll land into
master.Headline result — IMC Phototourism Grand Place Brussels (234 images)
vggt_sift_…_verified)vggt_omega_…_verified)Controlled A/B: the omega vs VGGT rows differ only in the per-cluster geometry model — same frontend,
partition, Fetzer, and retri.
How to review this efficiently
gtsfm/configs/vggt_sift_frontend_megaloc_phototourism_verified.yaml(every new flag is commented in place).scene_optimizer.py(+205) — the orchestration: global two-view verification → verifiedpartition → global Fetzer → post-merge retriangulation.
cluster_vggt_omega_with_frontend.py(+404) andvggt_omega_geometry_transformer.py(+347) are vendored from PR VGGT Omega implementation #1116 (Harneet) — review at theinterface level, not line-by-line. The load-bearing new logic is in
scene_optimizer.py,cluster_merging.py,cluster_vggt_with_frontend.py, andview_graph_calibration.py.New config flags (the feature switches — all default to the prior behavior)
use_verified_pipelinefalseuse_view_graph_calibrationfalseuse_global_view_graph_calibrationfalsecalibration_prior_focal_sigmareuse_global_correspondencesfalserecover_trackless_cameras_in_retriangulationfalseEvery flag off ⇒ byte-identical to the prior baseline. This is the key reviewer reassurance: the diff
is additive and gated.
Feature units → suggested clean branches (with landing order)
Dependency order: ① must land first (everything reads its
ClusterContextfields +use_verified_pipeline).②③④⑤ are independent of each other on top of ①. ⑥ is independent but its config overlays ①–⑤.
① Verified view-graph pipeline (foundation)
aeba561a,86a5a3ca,1c5ef76f,1dc132baMETIS on the verified subgraph (edges where
TwoViewResult.valid()), and adds a post-mergeretriangulation+BA stage (
results/merged_retriangulated/). Several follow-ups fix worker-OOM/dask-nestingby running the global frontend inline instead of via nested
worker_client()submission.scene_optimizer.py(orchestration),cluster_mvo.py,two_view_estimator.py(
create_two_view_results_inline),frontend/correspondence_generator/*(generate_correspondences_inline),cluster_optimizer_base.py(newClusterContextfields).② Global Fetzer focal calibration + focal-flow anchoring
e08be7c8,20adfc40,7b641621compute_global_view_graph_intrinsics),then anchors those focals through the cluster BA (σ=5px) and merge BA (σ=10px) so BAs optimize
poses, not focals (kills the focal/depth ambiguity that was diverging separator cameras across clusters).
graph_optimizer/view_graph_estimator/view_graph_calibration.py(+44),scene_optimizer.py,the
ba_options.calibration_prior_*plumbing.important for the omega arm (its anisotropic fx/fy is irrelevant; a single-focal
Cal3Bundleris produced).③ Peak frontend (config + PoseLib verifier)
170dc610,77796d20PoseLibVerifier(5-point + LO-RANSAC), 8192 ColmapSIFTkeypoints, 30/0.15 inlier gate — and retunes the Brussels baseline + verified configs.
configs/vggt_sift_frontend_megaloc_phototourism.yaml(+ verified),two_view_estimator.py,pyproject.toml(+poselib>=2.0).poselibdependency + verifier swap. Mostly config; lowest-risk unit.④ Global correspondence reuse (speedup)
1bc73192,ede5a9ba,8d6948bainstead of re-running its own frontend (which only cache-read the same data, serially + redundantly across
overlapping clusters). Pure speedup — per-edge
v_corris identical. (One commit,ede5a9ba, reverts anexperimental per-cluster triangulated-structure path that inflated clusters to 11k–15k tracks → OOM; it keeps
the focal-flow fix. See "reverted experiments" below.)
cluster_vggt_with_frontend.py(thereuse_global_correspondencesbranch),ClusterContext.global_v_corr_idxs_dict/global_keypoints./gcorrto the optimizer__repr__(cluster-cache key).⑤ Trackless-camera recovery
bbd79ff7(build-side "all-measurements"//allkpts),1af1e43e(retriangulation recovery)depth-lift build no longer lets zero-confidence VGGT depth veto a verified SIFT measurement (
/allkpts).(b) Cameras dropped at the root merge's track filter are captured and injected into the post-merge
retriangulation for a geometric (depth-independent) second chance; those that still can't triangulate are
cleanly dropped. Provably can't perturb the constructed cameras (recovered cams have <15 tracks ⇒
excluded from the retri BA factor graph).
cluster_merging.py(+38, capture +trackless_camerason the merge result),scene_optimizer.py(_run_post_merge_retriangulationinject + first-class diagnostic logging),cluster_vggt_with_frontend.py(build).follow-ups).
⑥ VGGT-Omega geometry integration
dc9824af(integration),7a7c51ca(preprocessing-dispatch fix)(
ClusterVGGTWithFrontendvia its injectedgeometry_transformer), not the bundledClusterVGGTOmegaWithFrontend(a master-based copy lacking Fetzer/reuse/recovery). Two small enabling changes:(a) guard
transformer.configaccess (omega has none) at__init__+__repr__; (b) generalize imagepreprocessing to follow the transformer — new
GeometryTransformer.load_image_batch(base = VGGT loader,omega overrides → its own 512/16-aligned loader). This is the cleanest standalone refactor in the PR and is a
no-op for all VGGT configs.
frontend/geometry_transformer.py(+20, the abstraction),frontend/vggt_omega_geometry_transformer.py(+347, vendored),cluster_optimizer/cluster_vggt_omega_with_frontend.py(+404, vendored),cluster_vggt.py(loader dispatch),cluster_optimizer/__init__.py(register),configs/vggt_omega_*(×2),.gitmodules+thirdparty/vggt-omegasubmodule,scripts/download_model_weights.sh,utils/torch.py.cc-by-nc-4.0, non-redistributable, CUDA-only.Submodule +
--hf_tokendownload required; omega module imports are lazy so non-omega paths/tests areunaffected. The geometry-transformer abstraction (a) is worth landing as its own tiny branch first —
omega then becomes a pure add-on.
Reverted experiments (intentionally not in the final pipeline)
use_triangulated_structure: true) — inflated clusters to 11k–15ktracks → 85–200s BAs → worker OOM. Reverted to VGGT depth-lift (
ede5a9ba); the flag remains but defaultsfalse.Known limitations / planned follow-ups (out of scope for this PR)
through the Sim3 merge unconstrained (one is ~90°-flipped). This is a pose-constraint problem →
PnP/resection, not a geometry-model one (omega didn't and couldn't fix it).
separate AUC lever.
How to run / verify
Drop
_omegafrom the config name for the VGGT arm. Metrics land inresults/merged_retriangulated/…json(
pose_auc_@3.0_deg,number_cameras_merged).