Fortress basin aoi - #81
Conversation
…new inference pipeline
Export FSC input cubes directly in the cell's native UTM CRS at scale=10 instead of round-tripping through EPSG:4326. The 4326 path left a grid- convergence tilt and a 0-valued border sliver around each tile, so adjacent cubes did not tile seamlessly. - eo_eval.py: new export_from_csv_utm_native — builds the request rectangle in the row's UTM crs, grows each cell by a buffer_m halo (neighbours overlap 2x buffer_m for a downstream mean-mosaic to reconcile), and exports at crs=<UTM>+scale=10 so EE resamples every band onto one shared 10m lattice. Verified: tiles are 108x108, origins on a common 10m grid, zero 0-border pixels, 80m overlap between neighbours, across all dates. - eo_eval.py: _export_for_polygon exports at scale=10 with the caller's crs (dropped an earlier crs_transform approach that failed with "inconsistent bounding boxes" on dates whose bands had mismatched native footprints). - build_aoi_cubes_gee_url.py: tile the AOI (Mode B), emit the GEE-UTM cube CSV, and drive the native export; --buffer-m controls the overlap halo. Assisted by Claude Code
url mode does a blocking getDownloadURL + download per cube, so it is network-bound and benefits from a small thread pool. Run the per-row exports through a ThreadPoolExecutor (default 4 workers); cloud/drive mode stays serial since it only fires batch submits that mutate the shared ee_task_list. - eo_eval.py: export_from_csv_utm_native gains max_workers (default 4); the per-row body is a closure dispatched via ThreadPoolExecutor for url mode. - build_aoi_cubes_gee_url.py: --max-workers flag wired through and logged. Workers are capped low deliberately: GEE throttles getDownloadURL, so a large pool returns 429s rather than speed. Assisted by Claude Code
…x duplication around grid management and reuse of existing cubes
|
Entrypoint into this PR is the following documentation :
As well as these 2 scripts :
I'm not strictly done with refactoring, but it's far enough along for you to try it out and see if the results line up with what you have on your end. No need to hurry, we can take a look together when I get back. Also, I'm pretty confident I'll be able to produce longer time series, as a single month's worth of fsc prediction for the Fortress Mountain Basin is less than 1% of my monthly quota on the non-commercial license, and takes a few hours to produce the cubes and run inference. |
marlens123
left a comment
marlens123
left a comment
There was a problem hiding this comment.
Most of it looks good to me!
I left some reoccurring comments about the supported UTM coordinate system - in case it's dynamic, some comments have to be changed.
The snippets in hybrid_inference_pipeline.md are working for me - except there is a change required for the GEE authentication on my end (see comment).
Things I haven't checked:
- the exported input and FSC tifs in more detail
- the aoi file
- the
PLAN.mdfile - the
test/files
I also do not fully understand/check the grid.py file and how exactly it tiles the aoi into cells.
2 participants
Summary
This PR adds a new data pipeline to enable processing AOIs directly instead of individual points, like was previously done to create the training datasets.
Changes
Description
Hybrid GEE→local pipeline
Allows FSC inference over an arbitrary AOI without a local raw archive: download cubes from GEE, infer locally.
Two new Typer CLIs in scripts/developer_scripts/fortress_mountain_basin/ :
build_aoi_cubes_gee_url.py: tiles AOI (Mode B lattice from the Local-data pipeline), creates cube CSV, downloads cubes via GEE URL mode.infer_aoi_cubes.py: reads cube CSV + cubes on disk, runs the finetuned model, writes daily FSC COGs for the AOI (1 per inference day for the whole AOI).Documentation can be found in docs/hybrid_inference_pipeline.md.
Seamless native-UTM GEE export
eo_eval.py: newexport_from_csv_utm_native. Builds the request rectangle directly in the row's UTM CRS atscale=10instead of round-tripping through 4326. Kills the grid-convergence 0-border seams between tiles.Parallelised via
ThreadPoolExecutorin url mode.Both UTM readers now accept
center_lat/lonorcenter_x/ydialects.Bug fix : missing normalizer (correctness, affects prior results)
_loader_bridge.pysetds.normalizer = None, feeding the encoder raw physical units (DN, dB, m, K) against a checkpoint trained on std-normalized inputs. Now builds the same Normalizer the GEE/eval path uses.Any FSC output produced (or that has not been re-run) before this commit is wrong-but-plausible.
Refactors
inference/model.py:build_model()extracted; both entry points had drifted private copies (one returned CPU model, one device-resident).inference/prebuilt.py:PrebuiltCubeSource: re-run inference over already-exported cubes, no re-export.driver.py: takesdays: list[date]instead ofwindow_start/window_end(non-contiguous dates now possible); adds read_workers threaded cube reads (~6.3→20.8 cubes/s at 4 workers).layout.py:cell_centre_lat_lon()+GEOGRAPHIC_CRShoisted out ofexporter.py(was private, driver reached intoexporter._cell_centre_lat_lon).grid.py:build_cells()splitfrom build_grid();build_cube_csv_for_gee_utm()adapter; filter_cells→_filter_cells.infer_bow_valley_daily_fsc.py:--cubes-only,--dates/--dates-file,--cube-dir, --read-workers.Current limitations
LLM Disclosure
This PR has been assisted by the use of Claude Code