Skip to content

Add a sensitivity check that runs one simulation per free parameter before a calibration #362

Description

@soaressgabriel

Description

A parameter reaches the simulation and may still have no effect on the compared variable, and the search then spends a dimension and its share of the population on it:

  • x, the flow recession coefficient, enters only the routed runoff arn (_dynamic_model.py#L535-L538); a calibration with --variable rnf searches it for nothing.
  • In a coupled run (Add the optional MODFLOW groundwater coupling #357) the baseflow comes from MODFLOW and alpha_gw has no effect; the documentation of the coupling tells the user to --fix it by hand.
  • A candidate can fail for a reason every candidate shares that the pre-checks cannot see (under the zones aggregation the station ids are not known before the run; a MODFLOW setup that never converges), and with the defaults the search spends 128 runs before the first generation line says so.

That the nine parameters are applied is guaranteed structurally, not by a check: the derived document writes them all (_worker.py#L286-L328), the format 1.0 model is strict with every field required, and the model reads each of them. What nothing says before the search is whether each one moves the compared series.

Proposed solution

  • rubem calibrate --sensitivity-check (sensitivity_check() in the runner, next to calibrate()) runs, in the pool of the calibration, the reference configuration (the starting point of the search) and one run per free parameter with that parameter moved by a fraction of its bound (a quarter of the range, toward the side with room), clipped to the bound and checked with is_admissible (moving w_1 up can leave w_3 negative; the weights are moved with w_3 derived). N + 1 runs, in parallel; the MODFLOW parameters named in --bound (Add the optional MODFLOW groundwater coupling #357) join like the others.
  • Each moved run is compared with the reference on the series the objective compares (aligned steps, the stations of the objective): maximum absolute difference and its value relative to the reference, plus the NSE of both. A parameter whose series is identical is reported as having no effect; a run that fails is reported with its error and does not end the check.
  • sensitivity.json in the run directory: the reference (parameters, NSE, station metrics), one entry per parameter (reference value, moved value, maximum absolute difference, relative difference, NSE, error), and the lists without_effect and failed. One log line per parameter and a closing warning naming the parameters without effect, with the advice to --fix them.
  • The command stops after the check (exit 0) and writes nothing else; the search is a separate invocation with the --fix the check suggested.

Tests that prove it: on the synthetic dataset x has no effect on rnf and an effect on arn; in a coupled configuration alpha_gw has no effect; the moved values are admissible; a moved run that fails is recorded and the check ends normally; sensitivity.json is written and the command exits 0.

Documentation: a "Sensitivity check" section in the calibration page, before "The search"; changelog.

Alternative solutions

Additional context

Part of the calibrator robustness work assessed on 2026-09-24. Independent of #358, #359, #360 and #361.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions