kemal/ashvale-station
1
0
Fork 0
mirror of https://github.com/lynchaos/ashvale-station.git synced 2026-09-12 12:47:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Earn the blend weights and the intervals from forecasts, not from refits

Browse Source 
fit() called learn(), and learn() updated three things: the RLS, the conformal
calibrator and the Hedge weights. Only the first belongs to a refit. The comment
above that loop already said so, and was wrong about what the code did.

Measured on 8.2 days of the live station, the 15 minute head had taken 977,078
Hedge updates from 758 distinct supervised pairs, a factor of 1,289, and the
1 day head 296,715 from 12 pairs, a factor of 24,726. A refit is not an outcome.
It is the same week of weather being read again, once every seven minutes.

Hedge is multiplicative, so an edge far too small to be real compounds to
certainty: twelve of twelve temperature and humidity heads had collapsed onto
climatology at a weight of 0.991 or above, while their own member_mae said the
members were within a few percent of each other. The ACI integrator moves by
gamma per observation, so it had likewise pinned against its clips, leaving the
6 hour temperature band (1.571 C) narrower than the 3 hour one (2.258 C), and
pressure at 1 day covering 3 of 7 with alpha jammed at the 0.005 floor.

Three changes, because fixing only the first would freeze the weights forever:

  - fit() calls refit_step(), which touches the regression and nothing else.
    The climatology and setpoint members were evaluated in that loop purely to
    feed the Hedge update, so fit() no longer needs a climatology or a
    setpoint_fn at all.
  - verify() feeds observe_outcome() with the member predictions the forecast
    was actually blended from. These are now written to the forecasts table at
    issue time, because the learned member cannot be recovered afterwards: the
    RLS has moved on.
  - a matured forecast teaches exactly once. It stays readable for an hour so
    the scorecard can aggregate a rolling window, which meant verify() was
    feeding the calibrator the same outcome about twelve times.

The Hedge weights additionally decline an outcome that overlaps the last one
they took, which is the stride rule from fit() applied on the scoring side.
Forecasts are issued every retrain tick, so at the 1 day horizon roughly two
hundred a day resolve against very nearly the same outcome. The conformal window
absorbs that, a quantile over duplicated scores being merely overconfident about
its sample size, but exponentiated gradient cannot.

Walk-forward over the full 8.2 day record, against the current code:

  mean MAE                0.856   (0.938 over the second half alone)
  heads improved          17/18
  beats persistence       7/18 -> 12/18
  coverage |dev from .90| 0.188 -> 0.060,  second half 0.112 -> 0.041

Decimating the conformal feed as well was measured and rejected. It reads better
(coverage |dev| 0.023) and is not: three heads fall below MIN_SCORES, drop to
1.645*sigma, and "cover" with a median band of +/- 107% relative humidity. At
h/4 and h/8 it never starves and lands within noise of not decimating at all, so
the simpler rule wins. Honest regressions: humidity at 12 hours is 19% worse,
and pressure past 6 hours is still under-covered, because at 1 day the point
forecast is genuinely poor and ACI can only widen so far.

A state file from before this change has its weights, member_mae, n_scored and
alpha reset on load. They are products of the replay, they are not evidence, and
they do not decay on their own: Hedge needs about twenty independent outcomes to
climb off its 1e-4 floor and the 1 d head sees one a day. The conformal scores
are kept, being residuals of roughly the right size, and the window refreshes
within about two days.

Schema migration verified against a pristine copy of the live database: 1,437
forecast rows preserved, five columns added, idempotent across restarts.
This commit is contained in:
kemal
2026-08-24 00:38:18 +01:00
parent 9a1033973d
commit 18b9a29ffa
6 changed files with 483 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ashvale/models/nowcast.py
+91 -36
View File
@@ -66,6 +66,8 @@ class ForecastHead:
self.eta = float(hedge_eta)
self.member_mae = np.zeros(len(MEMBERS))
self.n_scored = 0
# Validity time of the last outcome the Hedge weights learned from.
self.last_hedge_ts = -np.inf
# -------------------------------------------------------- prediction
@@ -91,26 +93,63 @@ class ForecastHead:
# ---------------------------------------------------------- learning
def learn(self, x: np.ndarray, anchor: float, truth: float,
climatology_delta: float = 0.0,
setpoint_delta: float = 0.0) -> float:
"""One supervised step given a matured target."""
deltas = np.array([0.0, float(climatology_delta),
float(self.model.predict(x)),
float(setpoint_delta)])
member_pred = anchor + deltas
def refit_step(self, x: np.ndarray, anchor: float, truth: float) -> None:
"""One regression update from a replayed historical pair.
This deliberately touches nothing but the RLS. The blend weights and
the conformal calibrator are statements about how this head's issued
forecasts actually turned out, and a refit is not an outcome: it is the
same week of weather being read again.
Measured on 8.2 days of real station data, the previous arrangement
(fit() calling a combined learn()) had put 977,078 Hedge updates through
the 15 minute head from 758 distinct supervised pairs, a factor of 1,289,
and 296,715 through the 1 day head from 12 pairs, a factor of 24,726.
Hedge is multiplicative, so an edge far too small to be real compounds
to certainty: twelve of twelve temperature and humidity heads had
collapsed onto climatology at a weight of 0.991 or above. The ACI
integrator, which moves by gamma per observation, had likewise pinned
against its clips, giving a 6 hour band narrower than the 3 hour one.
Feeding these two from verify() instead is worth 14.4% of MAE across
17 of 18 heads, and takes mean absolute coverage error from 0.188
to 0.059.
"""
self.model.update(x, truth - anchor)
def observe_outcome(self, members: np.ndarray, truth: float,
covered: Optional[bool] = None,
valid_ts: Optional[float] = None) -> float:
"""One matured forecast, scored against what actually happened.
`members` are the four point predictions this head issued, recovered
from the forecasts table. The learned one cannot be recomputed here
because the RLS has moved on since.
"""
member_pred = np.asarray(members, dtype=float)
losses = np.abs(member_pred - truth)
# Score the blend with the weights predict() would actually have used,
# before this sample's loss moves them. Doing it after is look-ahead:
# the residual handed to the conformal calibrator is then better than
# anything the forecaster can produce, so the intervals are calibrated
# about 2% too narrow. Coverage survived it only because ACI notices the
# extra misses and reopens the band, which is a correction that should
# not have been needed.
# Score the blend with the weights predict() actually used, before this
# outcome moves them. Doing it after is look-ahead: the residual handed
# to the conformal calibrator is then better than anything the
# forecaster can produce, so the intervals come out about 2% too narrow.
# Coverage survived it only because ACI notices the extra misses and
# reopens the band, which is a correction that should not be needed.
blended = float(np.dot(self.weights, member_pred))
residual = truth - blended
self.conformal.observe(residual)
self.conformal.observe(residual, covered=covered)
# The Hedge weights take an outcome only if it does not overlap the last
# one they took. Forecasts are issued every retrain tick, so at the 1 day
# horizon roughly two hundred of them per day resolve against what is
# very nearly the same outcome. The conformal window can absorb that,
# since a quantile over duplicated scores is merely over-confident about
# its sample size, but exponentiated gradient cannot: it would apply the
# same evidence two hundred times and saturate. This is the stride rule
# from fit(), applied on the scoring side.
if valid_ts is not None:
if valid_ts - self.last_hedge_ts < self.horizon_s:
return residual
self.last_hedge_ts = float(valid_ts)
# Hedge / exponentiated gradient on normalised losses.
#
@@ -129,8 +168,6 @@ class ForecastHead:
self.weights = np.clip(self.weights, 1e-4, None)
self.weights /= self.weights.sum()
self.model.update(x, truth - anchor)
self.member_mae = 0.98 * self.member_mae + 0.02 * losses
self.n_scored += 1
return residual
@@ -139,7 +176,9 @@ class ForecastHead:
return {"target": self.target, "horizon_s": self.horizon_s,
"model": self.model.to_dict(), "conformal": self.conformal.to_dict(),
"weights": self.weights.tolist(), "eta": self.eta,
"member_mae": self.member_mae.tolist(), "n_scored": self.n_scored}
"member_mae": self.member_mae.tolist(), "n_scored": self.n_scored,
"last_hedge_ts": (float(self.last_hedge_ts)
if np.isfinite(self.last_hedge_ts) else None)}
@classmethod
def from_dict(cls, s: Dict) -> "ForecastHead":
@@ -156,10 +195,34 @@ class ForecastHead:
w = np.ones(len(MEMBERS)) / len(MEMBERS)
h.weights = w
h.eta = s["eta"]
# A state file with no last_hedge_ts was written before the Hedge
# weights and the ACI integrator were cut off from fit()'s replay, so
# everything they hold is the product of the same week of weather read
# about a thousand times: weights pinned on one member at 0.99, alpha
# against a clip, member_mae an EMA over a million duplicated steps.
# None of that is evidence, and it does not decay on its own, because
# Hedge needs about twenty independent outcomes to climb back off the
# 1e-4 floor and the 1 d head sees one a day.
#
# The conformal scores are kept. They were also fed by the replay, so
# they are biased a little narrow, but they are absolute residuals of
# roughly the right size and the window refreshes within about two days
# of real outcomes. Clearing them instead would drop the long horizons
# onto 1.645*sigma for nine days, which is how you get a plus or minus
# of 115% relative humidity.
if "last_hedge_ts" not in s:
h.weights = np.ones(len(MEMBERS)) / len(MEMBERS)
h.member_mae = np.zeros(len(MEMBERS))
h.n_scored = 0
h.conformal.alpha = h.conformal.alpha_target
h.last_hedge_ts = -np.inf
return h
lh = s.get("last_hedge_ts")
h.last_hedge_ts = -np.inf if lh is None else float(lh)
mae = np.array(s["member_mae"], dtype=float)
# Same migration as the weights. Missing this one did not fail on load,
# it failed later inside learn() on a shape mismatch, which is a worse
# place to find out.
# it failed later inside the Hedge update on a shape mismatch, which is
# a worse place to find out.
if mae.size != len(MEMBERS):
mae = np.zeros(len(MEMBERS))
h.member_mae = mae
@@ -195,8 +258,7 @@ class NowcastEnsemble:
# ------------------------------------------------------------ train
def fit(self, X: np.ndarray, valid: np.ndarray, series: Dict[str, np.ndarray],
climatology=None, grid_ts: Optional[np.ndarray] = None,
passes: int = 1, max_pairs: int = 2500, setpoint_fn=None) -> Dict[str, int]:
passes: int = 1, max_pairs: int = 2500) -> Dict[str, int]:
"""Batch-update every head from history.
`max_pairs` bounds the work per head to the most recent samples.
@@ -216,8 +278,11 @@ class NowcastEnsemble:
# harmonics the record cannot yet resolve. The result was a six hour
# forecast of 53 C in a 24 C room, with a plus or minus of 0.43.
#
# The conformal calibrators and the Hedge weights are deliberately left
# alone: those are earned from scored forecasts, not from this regression.
# The conformal calibrators and the Hedge weights are left alone here,
# and refit_step is what enforces that. They are earned from scored
# forecasts in verify(), not from this regression. The climatology and
# setpoint members used to be evaluated in this loop purely to feed
# them, which is why this method no longer needs either.
for head in self.heads.values():
head.model.reset()
self.scaler.partial_fit(X[valid][:: max(1, X.shape[0] // 2000)])
@@ -235,15 +300,6 @@ class NowcastEnsemble:
if Xa.shape[0] > max_pairs:
Xa, dy, anchor = Xa[-max_pairs:], dy[-max_pairs:], anchor[-max_pairs:]
head = self.heads[(target, h)]
clim = np.zeros(Xa.shape[0])
if climatology is not None and grid_ts is not None and climatology.ready:
n = grid_ts.size
ts_a = grid_ts[:n - steps]
mask_len = min(ts_a.size, Xa.shape[0])
clim_now = climatology.predict(target, ts_a[-mask_len:])
clim_fut = climatology.predict(target, ts_a[-mask_len:] + h)
clim = np.zeros(Xa.shape[0])
clim[-mask_len:] = clim_fut - clim_now
# One pair per horizon, not one per grid row. Adjacent pairs at
# the 1 d horizon share 287 of their 288 samples, so training on
# every row hands the filter the same outcome 288 times and RLS
@@ -272,8 +328,7 @@ class NowcastEnsemble:
idx = idx[-max_pairs:]
for _ in range(max(int(passes), 1)):
for i in idx:
head.learn(Xa[i], anchor[i], anchor[i] + dy[i], clim[i],
setpoint_fn(target, h, anchor[i]) if setpoint_fn else 0.0)
head.refit_step(Xa[i], anchor[i], anchor[i] + dy[i])
counts[f"{target}@{h}"] = int(idx.size)
self.trained_rows = int(X.shape[0])
self.refit_phase += 1