Learner hygiene: Hedge loss scale, look-ahead residual, conformal minimum

Three changes to ForecastHead and the conformal calibrator, each measured
walk-forward on real data over seven train splits.

Hedge normalised its losses by the current sample's worst loss, so on a quiet
step where every member agreed to within 0.01 C whichever happened to be worst
still took the full exp(-eta) penalty, exactly as if it had been wrong by 5 C.
The regret bound assumes a fixed loss range, not a per-sample one, and the
symptom was weights that jumped around with no relation to horizon. Normalising
by the running member MAE instead is worth 1.81% of MAE, better on 106 of 126
heads, coverage unchanged.

The residual handed to the conformal calibrator was computed after this
sample's loss had already moved the weights, so it was better than anything the
forecaster could produce and the intervals were calibrated about 2% too narrow.
Coverage survived only because ACI notices the extra misses and reopens the
band, a correction that should never have been needed. Scoring the blend with
the pre-update weights leaves MAE untouched, as it must, and widens the
intervals 2% to the honest width.

The conformal quantile refused to produce a band below 20 scores. That number
is arbitrary: the (1-alpha) empirical quantile is the ceil((k+1)(1-alpha))-th
of k order statistics, so alpha = 0.10 needs 9. The 20 became actively harmful
in the previous commit but one, because striding pairs by the horizon leaves a
long-horizon head about 13 scores per refit. Twelve of eighteen heads therefore
fell through to 1.645*sigma with sigma from an unconstrained x'Px, giving bands
of +/- 45 C and +/- 115% relative humidity on a young station. Those cover, by
being absurd, which is why the backtest never flagged them: a long walk-forward
passes 20 scores early and never looks back. After the change all eighteen
heads have a band from the first fit, +/- 3.1 C and +/- 7.7% in the same place.

Also measured and deliberately not done: adding the Kalman level variance to
the predictive spread. It moves sigma by 0.06% at the shortest horizon and
0.00% everywhere else, so the plumbing to carry it through three files buys
nothing.

ashvale/models/nowcast.py

@@ -45,6 +45,11 @@ from .rls import AdaptiveConformal, RecursiveLeastSquares
 
 MEMBERS = ("persistence", "climatology", "learned", "setpoint")
 
+# Scored forecasts before the Hedge loss scale switches from this sample's
+# worst loss to the running member MAE. Until member_mae has seen anything it
+# is zeros, and dividing by that would hand every member the same penalty.
+_SCALE_WARMUP = 20
+
 
 class ForecastHead:
     """One target, one horizon."""
@@ -96,16 +101,34 @@ class ForecastHead:
         member_pred = anchor + deltas
         losses = np.abs(member_pred - truth)
 
-        # Hedge / exponentiated gradient on normalised losses
+        # Score the blend with the weights predict() would actually have used,
-        scale = max(float(np.max(losses)), 1e-6)
+        # before this sample's loss moves them. Doing it after is look-ahead:
-        self.weights *= np.exp(-self.eta * losses / scale)
+        # the residual handed to the conformal calibrator is then better than
-        self.weights = np.clip(self.weights, 1e-4, None)
+        # anything the forecaster can produce, so the intervals are calibrated
-        self.weights /= self.weights.sum()
+        # about 2% too narrow. Coverage survived it only because ACI notices the
-
+        # extra misses and reopens the band, which is a correction that should
+        # not have been needed.
         blended = float(np.dot(self.weights, member_pred))
         residual = truth - blended
         self.conformal.observe(residual)
 
+        # Hedge / exponentiated gradient on normalised losses.
+        #
+        # The scale must be a stable quantity, not this sample's worst loss.
+        # Dividing by max(losses) means that on a quiet step where every member
+        # agrees to within 0.01 C, whichever one happens to be worst still takes
+        # the full exp(-eta) penalty, exactly as if it had been wrong by 5 C, so
+        # the weights churn on noise. Hedge's regret bound assumes a fixed loss
+        # range. Normalising by the running member MAE instead is worth 1.8% of
+        # MAE across 106 of 126 heads on real data, with coverage unchanged.
+        if self.n_scored > _SCALE_WARMUP:
+            scale = max(float(np.mean(self.member_mae)), 1e-6)
+        else:
+            scale = max(float(np.max(losses)), 1e-6)
+        self.weights *= np.exp(-self.eta * losses / scale)
+        self.weights = np.clip(self.weights, 1e-4, None)
+        self.weights /= self.weights.sum()
+
         self.model.update(x, truth - anchor)
 
         self.member_mae = 0.98 * self.member_mae + 0.02 * losses

ashvale/models/rls.py

@@ -158,8 +158,22 @@ class AdaptiveConformal:
         self.scores: Deque[float] = deque(maxlen=int(window))
         self.hits: Deque[int] = deque(maxlen=int(window))
 
+    # Fewest scores from which a (1-alpha) empirical quantile exists at all.
+    # For alpha = 0.10 the band is the ceil(0.9*(k+1))-th of k order statistics,
+    # which needs k >= 9. Below that there is no quantile to take and the
+    # Gaussian fallback is the only option.
+    #
+    # This was 20, which is arbitrary and became actively harmful once training
+    # pairs were strided by the horizon: the long-horizon heads then earn about
+    # 13 scores per refit, so twelve of eighteen heads fell through to
+    # 1.645*sigma with sigma taken from an unconstrained x'Px. That produced
+    # bands of +/- 45 C and +/- 115% RH on a young station. They cover, being
+    # far too wide, but a plus or minus of 115% relative humidity is not a
+    # forecast.
+    MIN_SCORES = 9
+
     def quantile(self) -> float:
-        if len(self.scores) < 20:
+        if len(self.scores) < self.MIN_SCORES:
             return float("nan")
         a = float(np.clip(self.alpha, 0.005, 0.75))
         return float(np.quantile(np.asarray(self.scores), 1.0 - a, method="higher"))

tests/test_models.py

@@ -397,3 +397,63 @@ def test_loading_state_ignores_heads_this_build_no_longer_has():
     back.load_dict(saved)
     assert not any(t == "retired_signal" for t, _ in back.heads)
     assert len(back.heads) == len(cfg.targets) * len(cfg.horizons_s)
+
+
+def test_conformal_produces_a_band_from_the_fewest_scores_that_permit_one():
+    """20 was arbitrary and became harmful once pairs were strided.
+
+    The (1-alpha) empirical quantile is the ceil((k+1)(1-alpha))-th of k order
+    statistics, so alpha = 0.10 needs k >= 9. Requiring 20 threw away a valid
+    band at k = 13, which is roughly what a long-horizon head earns per refit
+    after striding, and dropped twelve of eighteen heads onto 1.645*sigma with
+    sigma from an unconstrained x'Px. That produced +/- 115% relative humidity.
+    """
+    from ashvale.models.rls import AdaptiveConformal
+
+    assert AdaptiveConformal.MIN_SCORES == 9
+
+    ac = AdaptiveConformal(alpha=0.10, gamma=0.01)
+    for i in range(8):
+        ac.observe(0.1 * (i + 1), True)
+    assert not np.isfinite(ac.quantile()), "8 scores cannot support a 90% band"
+
+    ac.observe(0.9, True)
+    q = ac.quantile()
+    assert np.isfinite(q), "9 scores must produce a band"
+    assert q > 0
+
+
+def test_strided_heads_do_not_fall_back_to_the_gaussian():
+    """The end-to-end version of the same thing, through a real fit."""
+    from ashvale.config import CONFIG
+    from ashvale.models.climatology import HarmonicClimatology
+    from ashvale.models.nowcast import NowcastEnsemble
+
+    rng = np.random.default_rng(19)
+    n = 700                                    # ~2.4 days, a young station
+    g = CONFIG.model.grid_s
+    ts = np.arange(n) * g + 1.7554e9
+    cols = {
+        "temperature": 22 + 3 * np.sin(np.arange(n) / 288.0) + 0.1 * rng.normal(size=n),
+        "humidity": 50 + 9 * np.cos(np.arange(n) / 288.0),
+        "pressure": 1013 + 2 * np.sin(np.arange(n) / 600.0),
+        "lux": np.clip(400 * np.sin(np.arange(n) / 288.0), 0, None),
+    }
+    X = rng.normal(size=(n, 33))
+    X[:, 0] = 1.0
+    valid = np.ones(n, dtype=bool)
+
+    clim = HarmonicClimatology(CONFIG.model.targets,
+                               min_days_annual=CONFIG.model.climatology_min_days_annual)
+    clim.fit(ts, {k: cols[k] for k in CONFIG.model.targets}, valid)
+    ens = NowcastEnsemble(CONFIG.model.targets, CONFIG.model.horizons_s, CONFIG.model)
+    ens.fit(X, valid, {k: cols[k] for k in CONFIG.model.targets}, clim, ts)
+
+    for (target, h), head in ens.heads.items():
+        q = head.conformal.quantile()
+        assert np.isfinite(q), (
+            f"{target}@{h}s has {len(head.conformal.scores)} scores and no band, "
+            "so it falls back to the Gaussian")
+        # and the band must be physical, not an unconstrained sigma
+        limit = {"temperature": 25.0, "humidity": 60.0, "pressure": 40.0}[target]
+        assert q < limit, f"{target}@{h}s band is +/- {q:.1f}, which is not a forecast"