Reported from a real station after 1.5 days: a six hour temperature forecast of
53 C in a 24 C room, and 9 C at one day, both carrying a plus or minus of 0.43.
Confidently wrong is the one failure this project is supposed to refuse.
Root cause. fit() replayed history into the live RLS on every retrain tick and
never reset, so 453 grid rows had produced 64,676 updates in a day and a half.
RLS with forgetting reads every update as fresh evidence, so the model believed
it had a hundred times the data it had: P collapsed, in-sample error looked
excellent, and the weights drifted without bound in directions the data never
excited. Measured: cond(P) 3.1e9 and ||theta|| 1680 against a median |theta| of
1.67. A refit now starts from the prior, which makes retraining idempotent.
Across 25 refits on the real data ||theta|| holds at 11.35, drifting 0.03, where
before it grew without limit.
The two largest weights were sin_doy and cos_doy at +1174 and +1191. Annual
harmonics were in the design matrix from the first sample, where they are
near-constant, near-collinear with each other and with the bias, and a
rank-deficient regressor is what RLS answers with enormous cancelling weights.
They are now held at zero until the record spans the same 120 days the
climatology fit already requires, because a day and a half of data says nothing
whatsoever about the season.
Also raised the standardiser's variance floor from 1e-8, which only caught a
bit-exactly constant column, to 1e-3. A feature that merely barely moves was
being divided by its own noise.
The conformal calibrators and Hedge weights are deliberately not reset by a
refit: those are earned from scored forecasts, not from this regression.
Backtest unchanged within noise, coverage still 89 to 91 across all 18 heads.
Four regression tests added, including that refitting the same history twice
must give the same model.
1. POST /api/recompute re-derives every compensated column from the untouched
raw values, removing the step a calibration otherwise leaves through the
history. Possible because temp_raw, cpu_temp and hum are never overwritten.
Idempotent by construction and tested per row: 0 of 6051 rows change on a
second run. 6069 rows in 0.25 s here, so a few seconds on the Pi.
2. Calibration now emits a 'discontinuity' event alongside the calibration log,
so downstream views can find the boundary without parsing prose.
3. Vendored Tailwind, Chart.js, hammer, the zoom plugin, KaTeX with its 20
woff2 faces, and both Google fonts into ashvale/static, served by the
station. 1.4 MB. Verified with every non-localhost request aborted in the
browser: zero external requests, equations still render, fonts still load.
The dashboard no longer needs internet.
4. 54 pytest cases over the pure numerics: physics closed forms and round
trips, both compensator inverse properties, the Kalman covariance
invariants and NIS consistency, the RLS trace cap under a deliberately
unexcited regressor, conformal coverage, and the Zambretti ordering. Wired
into CI after the seed step so the recompute cases have history. Writing
them caught my own sign error on the conformal update: a hit raises alpha
and narrows the band, which reads backwards until you follow it through.
5. Stats for Nerds gains the condition number of each head's covariance, a
standardised innovation histogram per Kalman filter from a bounded 600
sample ring buffer, and a reliability strip of realised against nominal
coverage. All arithmetic on data already in memory.
6. OutdoorProbe reads a DS18B20 over the kernel 1-Wire driver, no new
dependency. Polled on its own slower cadence because the sensor blocks for
up to 750 ms during conversion, which would eat a third of the 2 s sample
budget. Rejects the 85000 power-on sentinel and out-of-range values, and
reports age so a dead probe cannot masquerade as fresh.
kemal