p-phung/flood hazard logic

[p-phung]

AB#41515

Describe your changes

Introducing flood hazard logic:

Main script flood/forecast.py

Entry point for flood hazard logic via calculate_flood_forecasts(...)

1. Load input data (see Bronze input data below)

   • GloFAS NetCDF paths
   • Threshold JSON
   • Station-district mapping JSON
   • Population raster path
   • Flood extent raster paths

2. Build country spatial extent

   • Compute country bounding box from target admin areas.
   • Slice NetCDF files once to this bounding box.

3. Process discharge per station (for now 2 first stations for faster run)

   • Extract discharge ensemble values per lead time.
   • Derive lead-time severities from thresholds.
   • Skip stations with no threshold exceedance.

4. Build alert payload

   • Create one alert event per alerting station.
   • Add severity time-series data (run members + median).

5. Compute exposure

   • Select flood extent raster by highest matched return period.
   • Compute exposed population per place code.
   • Clip flood extent to impacted admin geometries.
   • Add admin area exposure and raster exposure to the event.

6. Write final output to local forecast folder

   • forecast.py fills DataSubmitter.
   • pipelines/infra/run_forecasts.py finalizes and writes forecast.json.
   • Default local base path is pipelines/output, resulting in paths like:
     • pipelines/output/floods/{ISO3}/{timestamp}/forecast.json
   • In this repository this appears under data/pipelines/output/floods/....

Supporting scripts

• extract_forecast.py

  • Samples GloFAS discharge values from (sliced) NetCDF rasters at station coordinates.
  • Produces per-station, per-lead-time ensemble discharge series.

• compute_alert_extent.py

  • Resolves the flood extent raster from flood extent raster catalog to use for an alert based on return period.
  • Falls back to closest lower return period, then to *_empty.tif.

• determine_exposure.py

  • Read mapping of station - admin areas.
  • Computes exposed population by intersecting flood extent with population raster.
  • Clips flood extent to affected admin areas for raster exposure output.

• utils_raster.py

  • Utility functions for geospatial preprocessing:
    • derive country bounding box from admin geometries,
    • slice NetCDF to country bounds,
    • get raster extent for output metadata.

Bronze input data

Data: link(~800MB). Extract and place in folder flood.

bronze/ holds preprocessed flood inputs used by forecast.py:

• bronze/glofas/

  • GloFAS discharge NetCDF files (ensemble forecast source).
  • Current code uses a path like dis_00_YYYYMMDDHH.nc and creates _sliced.nc files.

• bronze/thresholds/

  • Country-specific threshold JSON files (*_{ISO3}.json).
  • Used to convert discharge values into return-period-based alert severities.

• bronze/station-district/

  • Station-to-admin mapping JSON ({ISO3}_station_district_mapping.json).
  • Links station IDs to impacted place codes.

• bronze/population/

  • Country population raster ({ISO3}.tif).
  • Used to estimate exposed population inside flood extent.

• bronze/flood_extents/

  • Flood extent rasters by return period (flood_map_{ISO3}_rp*.tif) plus empty fallback.
  • Used to attach alert extent rasters to events.

Notes

• This logic is currently marked as temporary in places (TODO comments), especially around reading flood inputs directly from local bronze/ instead of fully through DataProvider, and station-basin-adminarea mapping (#Add match_station_basin function to identify unique basins for GloFAS stations #99)
• Currently a multiplier *100 is added in extracting GloFAS data, see extract_discharge_glofas_station() in flood/extract_forecast.py. This is to mock alert scenario. This will be removed when a scenario testing schema is in place.
• Tests in tests/ are generated by Claude.

Checklist before requesting a code review

• I have performed a self-review of my code
• I have addressed all Copilot comments
• I have asked the design team to review these changes, or: The changes do not touch the UI/UX
• I have added tests for my changes, or: Adding tests is unnecessary/irrelevant
• I have made sure that all automated checks pass before requesting a review
• I have updated all documentation where necessary
• I do not need any deviation from our PR guidelines

[p-phung]

@jannisvisser @gulfaraz :

• addressed your remain comments of the code
• for mock data data, indeed I was hesitant to upload to seed. the current zip file is about 800 including 1 single GloFAS nc file of ~700MB. I could use git lfs to commit instead of git only. But we will have another nc file(s) for trigger scenario which the size will be appx similar. Do we want GB of data inside the seed repo?
• for now, the data link the in the original PR message is updated and added to AB#41454
• I still haven't made test runs yet due to the API issue that was discussed with you --> to be continued.

[jannisvisser]

• you need to add pipelines/flood/bronze to data/.gitignore
• I notice the test data is only for ETH, which is fine for this item and this temporary data solution.
  • It does however mean that the main run command as defined in pipelines/README.md is failing (because the floods.yaml config is set to KEN). We need to align this. Let's switch the test-country from KEN to ETH throughout then? (I can do that). Or another solution? @gulfaraz what do you think?
  • Additionally, we need to switch to all countries, but let's do that in AB#41454 then. I made a note on it in refinement box.
• see some last inline comments
• functional testing, no-alert seems to work normal (no data posted), but for alert (with using *100)
  • when looking in the database I notice that all the population_exposed values per admin-area come out negative (below zero). Can you check? I have also made an add integrity checks task for sw-dev, as I think we should use such findings to improve on this.
  • also for alert-severity I see only one 'run' record (and one 'median) per alert, instead of 50. Can you check? (also added to list of 'add integrity checks', although not completely sure if we can implement it).

[p-phung]

@jannisvisser

• Negative population values: I found the issue. Cells with no data value (-99999) was assigned at the wrong order. These values were used for zonal_stats. Now we use the default no data value of the population raster file for consistency.
• Only one 'run' record (and one 'median) per alert, instead of 50, is expected. I fed one ensemble member (1 nc file) for now because feeding all 51 files will create 3.5GB which is too large. The glofas process is done in a for loop, which will go through all the nc files in this code line and will be loaded properly in later PBI.

old review