Learner hygiene: Hedge loss scale, look-ahead residual, conformal minimum

Three changes to ForecastHead and the conformal calibrator, each measured
walk-forward on real data over seven train splits.

Hedge normalised its losses by the current sample's worst loss, so on a quiet
step where every member agreed to within 0.01 C whichever happened to be worst
still took the full exp(-eta) penalty, exactly as if it had been wrong by 5 C.
The regret bound assumes a fixed loss range, not a per-sample one, and the
symptom was weights that jumped around with no relation to horizon. Normalising
by the running member MAE instead is worth 1.81% of MAE, better on 106 of 126
heads, coverage unchanged.

The residual handed to the conformal calibrator was computed after this
sample's loss had already moved the weights, so it was better than anything the
forecaster could produce and the intervals were calibrated about 2% too narrow.
Coverage survived only because ACI notices the extra misses and reopens the
band, a correction that should never have been needed. Scoring the blend with
the pre-update weights leaves MAE untouched, as it must, and widens the
intervals 2% to the honest width.

The conformal quantile refused to produce a band below 20 scores. That number
is arbitrary: the (1-alpha) empirical quantile is the ceil((k+1)(1-alpha))-th
of k order statistics, so alpha = 0.10 needs 9. The 20 became actively harmful
in the previous commit but one, because striding pairs by the horizon leaves a
long-horizon head about 13 scores per refit. Twelve of eighteen heads therefore
fell through to 1.645*sigma with sigma from an unconstrained x'Px, giving bands
of +/- 45 C and +/- 115% relative humidity on a young station. Those cover, by
being absurd, which is why the backtest never flagged them: a long walk-forward
passes 20 scores early and never looks back. After the change all eighteen
heads have a band from the first fit, +/- 3.1 C and +/- 7.7% in the same place.

Also measured and deliberately not done: adding the Kalman level variance to
the predictive spread. It moves sigma by 0.06% at the shortest horizon and
0.00% everywhere else, so the plumbing to carry it through three files buys
nothing.

Co-Authored-By: Claude Opus 5 <[email protected]>

ashvale/models/rls.py

@@ -158,8 +158,22 @@ class AdaptiveConformal:
         self.scores: Deque[float] = deque(maxlen=int(window))
         self.hits: Deque[int] = deque(maxlen=int(window))
 
+    # Fewest scores from which a (1-alpha) empirical quantile exists at all.
+    # For alpha = 0.10 the band is the ceil(0.9*(k+1))-th of k order statistics,
+    # which needs k >= 9. Below that there is no quantile to take and the
+    # Gaussian fallback is the only option.
+    #
+    # This was 20, which is arbitrary and became actively harmful once training
+    # pairs were strided by the horizon: the long-horizon heads then earn about
+    # 13 scores per refit, so twelve of eighteen heads fell through to
+    # 1.645*sigma with sigma taken from an unconstrained x'Px. That produced
+    # bands of +/- 45 C and +/- 115% RH on a young station. They cover, being
+    # far too wide, but a plus or minus of 115% relative humidity is not a
+    # forecast.
+    MIN_SCORES = 9
+
     def quantile(self) -> float:
-        if len(self.scores) < 20:
+        if len(self.scores) < self.MIN_SCORES:
             return float("nan")
         a = float(np.clip(self.alpha, 0.005, 0.75))
         return float(np.quantile(np.asarray(self.scores), 1.0 - a, method="higher"))