Three changes to ForecastHead and the conformal calibrator, each measured
walk-forward on real data over seven train splits.
Hedge normalised its losses by the current sample's worst loss, so on a quiet
step where every member agreed to within 0.01 C whichever happened to be worst
still took the full exp(-eta) penalty, exactly as if it had been wrong by 5 C.
The regret bound assumes a fixed loss range, not a per-sample one, and the
symptom was weights that jumped around with no relation to horizon. Normalising
by the running member MAE instead is worth 1.81% of MAE, better on 106 of 126
heads, coverage unchanged.
The residual handed to the conformal calibrator was computed after this
sample's loss had already moved the weights, so it was better than anything the
forecaster could produce and the intervals were calibrated about 2% too narrow.
Coverage survived only because ACI notices the extra misses and reopens the
band, a correction that should never have been needed. Scoring the blend with
the pre-update weights leaves MAE untouched, as it must, and widens the
intervals 2% to the honest width.
The conformal quantile refused to produce a band below 20 scores. That number
is arbitrary: the (1-alpha) empirical quantile is the ceil((k+1)(1-alpha))-th
of k order statistics, so alpha = 0.10 needs 9. The 20 became actively harmful
in the previous commit but one, because striding pairs by the horizon leaves a
long-horizon head about 13 scores per refit. Twelve of eighteen heads therefore
fell through to 1.645*sigma with sigma from an unconstrained x'Px, giving bands
of +/- 45 C and +/- 115% relative humidity on a young station. Those cover, by
being absurd, which is why the backtest never flagged them: a long walk-forward
passes 20 scores early and never looks back. After the change all eighteen
heads have a band from the first fit, +/- 3.1 C and +/- 7.7% in the same place.
Also measured and deliberately not done: adding the Kalman level variance to
the predictive spread. It moves sigma by 0.06% at the shortest horizon and
0.00% everywhere else, so the plumbing to carry it through three files buys
nothing.
Co-Authored-By: Claude Opus 5 <[email protected]>
The companion to the Kalman fix. RecursiveLeastSquares.from_dict and
AdaptiveConformal.from_dict restore lambda, delta, alpha, gamma and the
conformal window alongside their data, and load_dict replaces the
config-built heads with those, so every one of those knobs was immutable on
any station that already had state. Editing config.yaml and restarting looks
exactly like a change with no effect, which is the failure mode that cost real
time on the Kalman side of this before it was found.
Only the estimate is state now. Weights, covariances and conformal scores are
restored; tuning is re-applied from config on every load. The conformal deques
are rebuilt when the configured window changes, preserving their contents.
load_dict also skips heads for a target or horizon this build no longer has,
rather than resurrecting them from a stale file.
Found while implementing a damped-trend ensemble member, which was then
abandoned: see the following note.
Co-Authored-By: Claude Opus 5 <[email protected]>
fit() trained every head on every consecutive grid row. At the 1 d horizon on
a 5-minute grid adjacent pairs share 287 of their 288 samples, so the filter
was handed the same outcome 288 times and RLS with forgetting read each one as
fresh evidence:
horizon steps overlap independent events in a 400-score window
15m 3 66.7% 133.3
1h 12 91.7% 33.3
3h 36 97.2% 11.1
6h 72 98.6% 5.6
12h 144 99.3% 2.8
1d 288 99.7% 1.4
The day-ahead head was therefore fitted on roughly two independent outcomes by
a filter carrying 667 updates of memory, and its interval was a 90th percentile
of a sample of size one.
This is not a compute shortcut that trades accuracy for speed. Measured
walk-forward on four days of real station data and averaged over five train
splits, striding improves every horizon past fifteen minutes:
15m +0.6% 1h -12.2% 3h -31.7% 6h -33.3% 12h -39.5% 1d -14.4%
with coverage unchanged at 87 to 92%, and the fit 11.6x faster. The redundancy
was not merely wasted work, it was collapsing P onto the one direction the
repeated sample excited.
The stride phase rotates each refit and is persisted, so a long-lived station
eventually trains on every offset rather than seeing one sample in 288 forever,
and a restart does not pin it to phase 0. A floor relaxes the stride when a
long horizon on a short record would otherwise yield one or two pairs; 12 was
chosen by sweeping it across five splits rather than picked.
Single-split runs showed 10 to 17% regressions at the 1 d horizon that moved
with the parameter. Averaging over five splits removed them, which is the
expected result for a head fitted and scored on under two independent
outcomes. That horizon cannot be evaluated on a four-day record and was not
tuned against.
Incidentally, this also retires the parallel-retrain idea: the Pi's 42 s
retrain becomes a few seconds, and multiprocessing inside a 280 MB cap buys
nothing for a job that short.
Co-Authored-By: Claude Opus 5 <[email protected]>
Reported from a real station after 1.5 days: a six hour temperature forecast of
53 C in a 24 C room, and 9 C at one day, both carrying a plus or minus of 0.43.
Confidently wrong is the one failure this project is supposed to refuse.
Root cause. fit() replayed history into the live RLS on every retrain tick and
never reset, so 453 grid rows had produced 64,676 updates in a day and a half.
RLS with forgetting reads every update as fresh evidence, so the model believed
it had a hundred times the data it had: P collapsed, in-sample error looked
excellent, and the weights drifted without bound in directions the data never
excited. Measured: cond(P) 3.1e9 and ||theta|| 1680 against a median |theta| of
1.67. A refit now starts from the prior, which makes retraining idempotent.
Across 25 refits on the real data ||theta|| holds at 11.35, drifting 0.03, where
before it grew without limit.
The two largest weights were sin_doy and cos_doy at +1174 and +1191. Annual
harmonics were in the design matrix from the first sample, where they are
near-constant, near-collinear with each other and with the bias, and a
rank-deficient regressor is what RLS answers with enormous cancelling weights.
They are now held at zero until the record spans the same 120 days the
climatology fit already requires, because a day and a half of data says nothing
whatsoever about the season.
Also raised the standardiser's variance floor from 1e-8, which only caught a
bit-exactly constant column, to 1e-3. A feature that merely barely moves was
being divided by its own noise.
The conformal calibrators and Hedge weights are deliberately not reset by a
refit: those are earned from scored forecasts, not from this regression.
Backtest unchanged within noise, coverage still 89 to 91 across all 18 heads.
Four regression tests added, including that refitting the same history twice
must give the same model.
A room held at a setpoint is a different process from one left to drift. It is
a closed loop, and persistence, the baseline everything here is scored against,
is the wrong statement about it: the truth is not that it stays where it is, it
is that it returns to the setpoint.
So site.heating adds a fourth ensemble member, first order because that is what
a controlled system is:
dT_set(h) = (T_set - T_now) * (1 - exp(-h / tau))
Humidity follows and is the part that is easy to get wrong. Heating adds no
moisture, so vapour pressure is conserved and not relative humidity:
RH(h) = RH_now * es(T_now) / es(T_now + dT_set(h))
Warm the air and RH falls although nothing was dried, which is why a heated
house in winter is dry. The test asserts the dew point is unchanged to 1e-6.
Pressure gets zero: a thermostat cannot move the synoptic field.
Offered, not imposed. Hedge scores this member on realised error like any
other, so a wrong tau or a stale setpoint costs accuracy and gets down-weighted
rather than quietly biasing every forecast. Verified: on history with no
heating the ensemble assigned it weight 0.000. With heating off it returns zero
and is identical to persistence.
Going from three members to four means old saved heads must migrate.
from_dict reinitialises weights and member_mae. I missed member_mae first time
and it did not fail on load, it failed later inside learn() on a broadcast
error, which is a much worse place to find out; the migration test now covers
both and calls learn() to prove it.
Settings tab gains the toggle, setpoint and time constant. Turning heating on
or off is treated as a regime change like a door: discontinuity marker plus a
queued retrain.
kemalandClaude Opus 5