Watershed Modeling Python for hydrology automation

Split-Sample Validation for Hydrologic Models

A calibrated model reproduces the period it was calibrated on; that is what calibration means and it is not evidence of anything. The question that matters is whether the parameters describe the catchment or the period, and only data the optimiser never saw can answer it. This guide covers the split-sample designs that do, as part of the model calibration and objective functions topic within rainfall-runoff modeling and hydrologic simulation.

Prerequisites

  • An observed flow record long enough to split — a decade at minimum, with several years in each half.
  • A calibration driver that can be pointed at a date range, and forcing covering the whole record.
  • pandas, numpy, and the metric implementations from computing Nash-Sutcliffe efficiency and KGE in Python.

Core Technique: Four Split Designs

Four Ways to Split One Record Simple split calibrates on the first half and validates on the second. Reversed split does the opposite and should give a similar answer. Differential split calibrates on wet years and validates on dry ones. Three-way split adds a test period that is touched exactly once. simple calibrate validate reversed validate calibrate differential split by wetness, not by date — wet years calibrate, dry years validate three-way calibrate develop test, once Use the three-way design whenever the model will be revised after seeing validation results.

The reversed split is the cheapest useful addition and the most often skipped. Running both directions and comparing tells you whether the two halves are hydrologically similar: if calibrating on either half validates well on the other, the parameters describe the catchment. If one direction works and the other does not, one half contains behaviour the other does not, and that is a finding about the record.

Annotated Code Example

python 
import logging
from dataclasses import dataclass

import numpy as np
import pandas as pd

log = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")


@dataclass
class SplitResult:
    design: str
    calibration_period: tuple
    validation_period: tuple
    calibration_score: float
    validation_score: float

    @property
    def drop(self) -> float:
        return self.calibration_score - self.validation_score


def kge(sim: np.ndarray, obs: np.ndarray) -> float:
    """Kling-Gupta efficiency on aligned, gap-free arrays."""
    m = np.isfinite(sim) & np.isfinite(obs)
    s, o = sim[m], obs[m]
    if len(s) < 10 or o.std() == 0:
        return float("nan")
    r = float(np.corrcoef(s, o)[0, 1])
    alpha = float(s.std() / o.std())
    beta = float(s.mean() / o.mean())
    return float(1 - np.sqrt((r - 1) ** 2 + (alpha - 1) ** 2 + (beta - 1) ** 2))


def split_sample(
    observed: pd.Series,
    calibrate_fn,
    simulate_fn,
    design: str = "simple",
    warmup_days: int = 365,
) -> list[SplitResult]:
    """
    Run a split-sample validation.

    Parameters
    ----------
    observed     : Observed discharge, indexed by date.
    calibrate_fn : Callable(period) -> parameter dict.
    simulate_fn  : Callable(params, period) -> simulated Series.
    design       : "simple", "reversed", "both", or "differential".
    warmup_days  : Days discarded at the start of every simulated period, so
                   the initial state does not contaminate the score.
    """
    obs = observed.dropna().sort_index()
    if len(obs) < 3 * 365:
        raise ValueError("record is too short to split meaningfully")

    def score(params, period):
        sim = simulate_fn(params, period)
        start = period[0] + pd.Timedelta(days=warmup_days)
        # Discarding the warm-up is not optional: a model started from an
        # arbitrary state reproduces the first months badly no matter how
        # well calibrated, and including them penalises every period equally
        # but unequally in magnitude.
        s = sim.loc[start:period[1]]
        o = obs.loc[start:period[1]]
        common = s.index.intersection(o.index)
        return kge(s.loc[common].to_numpy(), o.loc[common].to_numpy())

    mid = obs.index[len(obs) // 2]
    first = (obs.index[0], mid)
    second = (mid, obs.index[-1])

    designs = []
    if design in ("simple", "both"):
        designs.append(("simple", first, second))
    if design in ("reversed", "both"):
        designs.append(("reversed", second, first))
    if design == "differential":
        # Split by annual total rather than by date, so the test is whether the
        # parameters transfer across wetness regimes.
        annual = obs.resample("YE").sum()
        median = annual.median()
        wet_years = set(annual[annual >= median].index.year)
        wet_mask = obs.index.year.isin(list(wet_years))
        log.info("Differential split: %d wet year(s), %d dry year(s)",
                 len(wet_years), len(annual) - len(wet_years))
        designs.append(("differential (wet→dry)",
                        (obs.index[0], obs.index[-1]), (obs.index[0], obs.index[-1])))
        # Where the calibrate/simulate callables accept a mask, pass it; the
        # period tuples above are retained only for reporting.

    results = []
    for name, cal_period, val_period in designs:
        params = calibrate_fn(cal_period)
        cal = score(params, cal_period)
        val = score(params, val_period)
        r = SplitResult(name, cal_period, val_period, cal, val)
        results.append(r)

        log.info("%-22s calibration KGE %.3f, validation KGE %.3f (drop %.3f)",
                 name, cal, val, r.drop)
        if r.drop > 0.20:
            log.error("A drop of %.2f suggests the parameters absorbed "
                      "period-specific behaviour rather than catchment behaviour",
                      r.drop)
        elif r.drop < -0.05:
            log.warning("Validation scored HIGHER than calibration by %.2f — the "
                        "validation period is probably easier, not the model "
                        "better", -r.drop)
        else:
            log.info("Drop of %.2f is within the range expected from fitting", r.drop)

    if len(results) == 2:
        asym = abs(results[0].validation_score - results[1].validation_score)
        if asym > 0.15:
            log.error("The two directions disagree by %.2f KGE — the halves of "
                      "the record are hydrologically different; report both",
                      asym)
        else:
            log.info("Both directions agree to within %.2f KGE", asym)
    return results


# --- Example usage ---
# results = split_sample(observed_q, calibrate_fn=my_calibrator,
#                        simulate_fn=my_model, design="both")

Parameter Reference

Choice Recommendation Why
Split point Halfway by time, or by wetness A split at a land-use change is a different test, worth doing deliberately
warmup_days 365 for a continuous model An arbitrary initial state contaminates the first months
Design both at minimum The reversed direction is nearly free and often the informative one
Acceptable drop ≤ 0.10 in KGE Above 0.20, treat the calibration as overfitted
Recalibration after validation Only with a three-way split Otherwise the validation period becomes a calibration period

Worked Example: Reading the Two Directions

A 22-year record on a 480 km² basin:

Design Calibration KGE Validation KGE Drop
Simple (first → second) 0.84 0.79 0.05
Reversed (second → first) 0.81 0.62 0.19
Differential (wet → dry) 0.86 0.54 0.32

Read as a set, these say something specific. The two chronological directions disagree, and the differential test fails badly. The parameters calibrated on wet conditions do not reproduce dry ones, which points at the storage or recession components rather than at the event response. That is a diagnosis, and it is only available because three tests were run rather than one.

Reporting the simple split alone — 0.84 calibration, 0.79 validation — would have described this model as transferable.

One Split Says Transferable, Three Say Otherwise Under the simple split calibration KGE is 0.84 and validation 0.79. Reversed, calibration is 0.81 and validation 0.62. Under the differential wet-to-dry split, calibration is 0.86 and validation 0.54, a drop of 0.32. 0.4 0.6 0.8 1.0 KGE simple drop 0.05 reversed drop 0.19 differential drop 0.32 calibration validation

Warm-up length matters more on a model with slow stores than on a flashy one, and the score keeps improving until the state has forgotten its initial value.

How Long the Warm-Up Needs to Be Validation KGE against discarded warm-up days. A fast-response model reaches its plateau after about 60 days. A model with a slow groundwater store keeps improving until about 400 days, so a one-year warm-up is the safe default. fast-response model — settled by 60 days slow groundwater store — still rising at 200 0 60 120 200 365 550 warm-up days discarded before scoring validation KGE

Gotchas and Edge Cases

  • Warm-up included in the score. Every period is penalised, but the calibration period’s optimiser compensates for it and the validation period’s does not, which exaggerates the drop.
  • A split at a regime change. Calibrating before a dam and validating after tests something real but not what the report usually claims. Split deliberately and say so.
  • Recalibrating after a bad validation. The validation period becomes a calibration period the moment it influences a parameter.
  • Only the forward direction reported. Half the information for a fraction of the extra cost.
  • Validation higher than calibration read as success. It nearly always means the validation period is easier — fewer events, or events of a kind the model handles well.
  • Both halves too short. With five years each, the split is dominated by which half got the big flood. A decade per half is a reasonable floor.
  • One objective function throughout. A model that transfers on KGE and fails on log-NSE transfers for peaks and not for baseflow, which is worth knowing.