Skip to content

Let a calibration stop on a tolerance, on a stalled efficiency and on a generation that failed entirely #360

Description

@soaressgabriel

Description

The search stops only when SciPy's convergence test fires or --maxiter is reached. Three things a person watching a calibration needs are missing, and the reason the search ended is not recorded in its own right:

  • tol and atol of differential_evolution are not exposed; CalibrationSettings has neither (runner.py#L150-L237) and every search runs with the 1 % default the documentation describes ("The evaluation budget").
  • The per-generation callback logs and never stops (runner.py#L346-L376). A search whose best NSE has not moved for twenty generations keeps spending a population of model runs per generation until --maxiter.
  • A configuration whose every evaluation fails (a station mismatch the pre-checks cannot see under the zones aggregation, a MODFLOW setup that never converges) spends the whole budget before the runner reports "Every evaluation of the calibration failed" (runner.py#L586-L590). With the defaults that is 12 928 runs to learn nothing.
  • result.json records only SciPy's message and success; a stop asked by the callback gives success=False and the generic "callback function requested stop early".

Two SciPy facts shape the design. INADMISSIBLE_OBJECTIVE is finite, 1e30 (objective.py#L43-L52), so one failed member blows up the standard deviation of the population energies and the tol test cannot fire while any evaluation of the generation failed: whole-generation failure needs a rule of its own. And the callback first fires after generation 1: the initial population goes through the workers map before it (runner.py#L552), so a failure of the whole initial population is seen by the map, not by the callback.

Proposed solution

  • --tol (default 0.01) and --atol (default 0), passed through as tol and atol and recorded in settings.
  • --stall-generations K (default 0, off) and --stall-min-delta-nse D (default 0.001, read only with K): the callback keeps the best NSE of every generation, from intermediate_result.fun (NSE = 1 - sqrt(f / 1000) / 100, no records re-read), and stops the search when it improved by less than D over the last K generations.
  • A generation whose every evaluation failed (errors other than inadmissible) stops the search: a wrapper around executor.map reads the records of the batch it just ran and raises a dedicated exception; calibrate consolidates and raises the existing CalibrationError with the first error, after one population instead of the whole budget. Because the wrapper sees the initial population, a hopeless configuration costs one population.
  • termination (from Write the result of a calibration that ends before the search finishes #359) gains converged (SciPy's tolerance), maxiter and stalled (with the K and D that fired). success keeps the optimizer's flag; the documentation says termination is the field to read.
  • The generation line gains the number of failed evaluations of the generation and the time since the previous line.

Tests that prove it: tol and atol reach the search arguments (the "documented arguments" test); a stall stop on an objective that stops improving ends with termination == "stalled" after K generations without a gain; a configuration whose every evaluation fails ends after the initial population (nfev equal to the population size) with the existing message; the generation line.

Documentation: "Settings", the convergence paragraph of "The evaluation budget", the result.json field list, "Progress on the terminal"; changelog.

Alternative solutions

  • Stop on stalled objective instead of NSE. The objective is a monotone function of the mean NSE, so the two are the same rule; NSE is the unit the user reads.
  • Detect the failed generation in the callback. Rejected: it fires only after generation 1, so the initial population would be spent twice over before the first check.

Additional context

Part of the calibrator robustness work assessed on 2026-09-24. Depends on #359 for the termination field; touches _Progress, which #361 touches too (rebase, not a dependency).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions