mirror of
https://github.com/lynchaos/ashvale-station.git
synced 2026-09-12 12:47:49 +00:00
18b9a29ffa1fc2557302aefedbe30bda3526dd9b
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 kemal
|
18b9a29ffa |
Earn the blend weights and the intervals from forecasts, not from refits
fit() called learn(), and learn() updated three things: the RLS, the conformal
calibrator and the Hedge weights. Only the first belongs to a refit. The comment
above that loop already said so, and was wrong about what the code did.
Measured on 8.2 days of the live station, the 15 minute head had taken 977,078
Hedge updates from 758 distinct supervised pairs, a factor of 1,289, and the
1 day head 296,715 from 12 pairs, a factor of 24,726. A refit is not an outcome.
It is the same week of weather being read again, once every seven minutes.
Hedge is multiplicative, so an edge far too small to be real compounds to
certainty: twelve of twelve temperature and humidity heads had collapsed onto
climatology at a weight of 0.991 or above, while their own member_mae said the
members were within a few percent of each other. The ACI integrator moves by
gamma per observation, so it had likewise pinned against its clips, leaving the
6 hour temperature band (1.571 C) narrower than the 3 hour one (2.258 C), and
pressure at 1 day covering 3 of 7 with alpha jammed at the 0.005 floor.
Three changes, because fixing only the first would freeze the weights forever:
- fit() calls refit_step(), which touches the regression and nothing else.
The climatology and setpoint members were evaluated in that loop purely to
feed the Hedge update, so fit() no longer needs a climatology or a
setpoint_fn at all.
- verify() feeds observe_outcome() with the member predictions the forecast
was actually blended from. These are now written to the forecasts table at
issue time, because the learned member cannot be recovered afterwards: the
RLS has moved on.
- a matured forecast teaches exactly once. It stays readable for an hour so
the scorecard can aggregate a rolling window, which meant verify() was
feeding the calibrator the same outcome about twelve times.
The Hedge weights additionally decline an outcome that overlaps the last one
they took, which is the stride rule from fit() applied on the scoring side.
Forecasts are issued every retrain tick, so at the 1 day horizon roughly two
hundred a day resolve against very nearly the same outcome. The conformal window
absorbs that, a quantile over duplicated scores being merely overconfident about
its sample size, but exponentiated gradient cannot.
Walk-forward over the full 8.2 day record, against the current code:
mean MAE 0.856 (0.938 over the second half alone)
heads improved 17/18
beats persistence 7/18 -> 12/18
coverage |dev from .90| 0.188 -> 0.060, second half 0.112 -> 0.041
Decimating the conformal feed as well was measured and rejected. It reads better
(coverage |dev| 0.023) and is not: three heads fall below MIN_SCORES, drop to
1.645*sigma, and "cover" with a median band of +/- 107% relative humidity. At
h/4 and h/8 it never starves and lands within noise of not decimating at all, so
the simpler rule wins. Honest regressions: humidity at 12 hours is 19% worse,
and pressure past 6 hours is still under-covered, because at 1 day the point
forecast is genuinely poor and ACI can only widen so far.
A state file from before this change has its weights, member_mae, n_scored and
alpha reset on load. They are products of the replay, they are not evidence, and
they do not decay on their own: Hedge needs about twenty independent outcomes to
climb off its 1e-4 floor and the 1 d head sees one a day. The conformal scores
are kept, being residuals of roughly the right size, and the window refreshes
within about two days.
Schema migration verified against a pristine copy of the live database: 1,437
forecast rows preserved, five columns added, idempotent across restarts.
|
||
|
kemal
|
f1647788c8 |
Learner hygiene: Hedge loss scale, look-ahead residual, conformal minimum
Three changes to ForecastHead and the conformal calibrator, each measured walk-forward on real data over seven train splits. Hedge normalised its losses by the current sample's worst loss, so on a quiet step where every member agreed to within 0.01 C whichever happened to be worst still took the full exp(-eta) penalty, exactly as if it had been wrong by 5 C. The regret bound assumes a fixed loss range, not a per-sample one, and the symptom was weights that jumped around with no relation to horizon. Normalising by the running member MAE instead is worth 1.81% of MAE, better on 106 of 126 heads, coverage unchanged. The residual handed to the conformal calibrator was computed after this sample's loss had already moved the weights, so it was better than anything the forecaster could produce and the intervals were calibrated about 2% too narrow. Coverage survived only because ACI notices the extra misses and reopens the band, a correction that should never have been needed. Scoring the blend with the pre-update weights leaves MAE untouched, as it must, and widens the intervals 2% to the honest width. The conformal quantile refused to produce a band below 20 scores. That number is arbitrary: the (1-alpha) empirical quantile is the ceil((k+1)(1-alpha))-th of k order statistics, so alpha = 0.10 needs 9. The 20 became actively harmful in the previous commit but one, because striding pairs by the horizon leaves a long-horizon head about 13 scores per refit. Twelve of eighteen heads therefore fell through to 1.645*sigma with sigma from an unconstrained x'Px, giving bands of +/- 45 C and +/- 115% relative humidity on a young station. Those cover, by being absurd, which is why the backtest never flagged them: a long walk-forward passes 20 scores early and never looks back. After the change all eighteen heads have a band from the first fit, +/- 3.1 C and +/- 7.7% in the same place. Also measured and deliberately not done: adding the Kalman level variance to the predictive spread. It moves sigma by 0.06% at the shortest horizon and 0.00% everywhere else, so the plumbing to carry it through three files buys nothing. |
||
|
kemal
|
41a46ae14a |
Take model tuning from config too, not from the saved state
The companion to the Kalman fix. RecursiveLeastSquares.from_dict and AdaptiveConformal.from_dict restore lambda, delta, alpha, gamma and the conformal window alongside their data, and load_dict replaces the config-built heads with those, so every one of those knobs was immutable on any station that already had state. Editing config.yaml and restarting looks exactly like a change with no effect, which is the failure mode that cost real time on the Kalman side of this before it was found. Only the estimate is state now. Weights, covariances and conformal scores are restored; tuning is re-applied from config on every load. The conformal deques are rebuilt when the configured window changes, preserving their contents. load_dict also skips heads for a target or horizon this build no longer has, rather than resurrecting them from a stale file. Found while implementing a damped-trend ensemble member, which was then abandoned: see the following note. |
||
|
kemal
|
e3176e29c9 |
Stride training pairs by the horizon instead of by the grid row
fit() trained every head on every consecutive grid row. At the 1 d horizon on
a 5-minute grid adjacent pairs share 287 of their 288 samples, so the filter
was handed the same outcome 288 times and RLS with forgetting read each one as
fresh evidence:
horizon steps overlap independent events in a 400-score window
15m 3 66.7% 133.3
1h 12 91.7% 33.3
3h 36 97.2% 11.1
6h 72 98.6% 5.6
12h 144 99.3% 2.8
1d 288 99.7% 1.4
The day-ahead head was therefore fitted on roughly two independent outcomes by
a filter carrying 667 updates of memory, and its interval was a 90th percentile
of a sample of size one.
This is not a compute shortcut that trades accuracy for speed. Measured
walk-forward on four days of real station data and averaged over five train
splits, striding improves every horizon past fifteen minutes:
15m +0.6% 1h -12.2% 3h -31.7% 6h -33.3% 12h -39.5% 1d -14.4%
with coverage unchanged at 87 to 92%, and the fit 11.6x faster. The redundancy
was not merely wasted work, it was collapsing P onto the one direction the
repeated sample excited.
The stride phase rotates each refit and is persisted, so a long-lived station
eventually trains on every offset rather than seeing one sample in 288 forever,
and a restart does not pin it to phase 0. A floor relaxes the stride when a
long horizon on a short record would otherwise yield one or two pairs; 12 was
chosen by sweeping it across five splits rather than picked.
Single-split runs showed 10 to 17% regressions at the 1 d horizon that moved
with the parameter. Averaging over five splits removed them, which is the
expected result for a head fitted and scored on under two independent
outcomes. That horizon cannot be evaluated on a four-day record and was not
tuned against.
Incidentally, this also retires the parallel-retrain idea: the Pi's 42 s
retrain becomes a few seconds, and multiprocessing inside a 280 MB cap buys
nothing for a job that short.
|
||
|
kemal
|
485affe956 |
Fix the runaway forecasts: refits accumulated, and annual terms fitted too early
Reported from a real station after 1.5 days: a six hour temperature forecast of 53 C in a 24 C room, and 9 C at one day, both carrying a plus or minus of 0.43. Confidently wrong is the one failure this project is supposed to refuse. Root cause. fit() replayed history into the live RLS on every retrain tick and never reset, so 453 grid rows had produced 64,676 updates in a day and a half. RLS with forgetting reads every update as fresh evidence, so the model believed it had a hundred times the data it had: P collapsed, in-sample error looked excellent, and the weights drifted without bound in directions the data never excited. Measured: cond(P) 3.1e9 and ||theta|| 1680 against a median |theta| of 1.67. A refit now starts from the prior, which makes retraining idempotent. Across 25 refits on the real data ||theta|| holds at 11.35, drifting 0.03, where before it grew without limit. The two largest weights were sin_doy and cos_doy at +1174 and +1191. Annual harmonics were in the design matrix from the first sample, where they are near-constant, near-collinear with each other and with the bias, and a rank-deficient regressor is what RLS answers with enormous cancelling weights. They are now held at zero until the record spans the same 120 days the climatology fit already requires, because a day and a half of data says nothing whatsoever about the season. Also raised the standardiser's variance floor from 1e-8, which only caught a bit-exactly constant column, to 1e-3. A feature that merely barely moves was being divided by its own noise. The conformal calibrators and Hedge weights are deliberately not reset by a refit: those are earned from scored forecasts, not from this regression. Backtest unchanged within noise, coverage still 89 to 91 across all 18 heads. Four regression tests added, including that refitting the same history twice must give the same model. |
||
|
kemal
|
98210bff8f |
Six enhancements: recompute, markers, vendoring, tests, nerd stats, DS18B20
1. POST /api/recompute re-derives every compensated column from the untouched raw values, removing the step a calibration otherwise leaves through the history. Possible because temp_raw, cpu_temp and hum are never overwritten. Idempotent by construction and tested per row: 0 of 6051 rows change on a second run. 6069 rows in 0.25 s here, so a few seconds on the Pi. 2. Calibration now emits a 'discontinuity' event alongside the calibration log, so downstream views can find the boundary without parsing prose. 3. Vendored Tailwind, Chart.js, hammer, the zoom plugin, KaTeX with its 20 woff2 faces, and both Google fonts into ashvale/static, served by the station. 1.4 MB. Verified with every non-localhost request aborted in the browser: zero external requests, equations still render, fonts still load. The dashboard no longer needs internet. 4. 54 pytest cases over the pure numerics: physics closed forms and round trips, both compensator inverse properties, the Kalman covariance invariants and NIS consistency, the RLS trace cap under a deliberately unexcited regressor, conformal coverage, and the Zambretti ordering. Wired into CI after the seed step so the recompute cases have history. Writing them caught my own sign error on the conformal update: a hit raises alpha and narrows the band, which reads backwards until you follow it through. 5. Stats for Nerds gains the condition number of each head's covariance, a standardised innovation histogram per Kalman filter from a bounded 600 sample ring buffer, and a reliability strip of realised against nominal coverage. All arithmetic on data already in memory. 6. OutdoorProbe reads a DS18B20 over the kernel 1-Wire driver, no new dependency. Polled on its own slower cadence because the sensor blocks for up to 750 ms during conversion, which would eat a third of the 2 s sample budget. Rejects the 85000 power-on sentinel and out-of-range values, and reports age so a dead probe cannot masquerade as fresh. |