mirror of
https://github.com/lynchaos/ashvale-station.git
synced 2026-09-12 12:47:49 +00:00
9a1033973dffa53827e18983e452d0c168ee2ebe
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 kemal
|
9a1033973d |
Treat the board being moved as a regime change, from the fused IMU attitude
The accelerometer and gyroscope were logged and never used. They measure nothing about weather, but they do measure the one thing about this station that nothing else can see: whether the sensor is still where it was. Measured over four and a half days on the real station, four genuine movements each stepped the temperature by a median of 1.02 C, against an ordinary fifteen minute change of 0.107 C with a 95th percentile of 0.841. A move therefore lands past the 95th percentile of normal variation. The heads carry about 55 hours of memory, so an undeclared move contaminates two days of training with a discontinuity they will try to fit rather than ignore. This now gets the same treatment set_environment gives a window being opened, because it is the same event: the coupling between the sensor and what it is measuring changed, and nothing in the data says so. Three choices in here were made by measurement, and the obvious one was wrong. Raw accelerometer looks like the natural input and is not. Over the same record a gravity-vector detector fires 112 times against this one's 4, because RTIMULib's gyro fusion removes exactly the desk vibration a bare accelerometer picks up. The fused pitch and roll have a p99 sample-to-sample noise of 0.0001 degrees, so a one degree trigger carries four decades of headroom. Yaw and compass are excluded. They are the only attitude outputs that depend on the magnetometer, and indoors the magnetometer is measuring the building. RTIMULib restarts its fusion from a default attitude when SenseHat is reconstructed, which put an 18 degree step in the record on every one of this station's seven service restarts. Without a settle window every deploy would queue a retrain. 300 seconds rather than 180: one artifact appeared three minutes after a restart, still converging. Replayed against the full record the detector finds 4 genuine movements and leaks 0 artifacts. |
||
|
kemal
|
3dd45f7ebf |
Joystick labelling, a clock guard, and throttle logging
Three things the hardware offers that the code ignored. The joystick has never had a line of code. Left records a dry label, right a wet one, middle cycles the LED scene, and a full-panel flash acknowledges the press because a headless box gives no other sign and a button you cannot tell worked gets pressed twice. Precipitation is the weakest head in the bank and strong labels are its binding constraint: this station has 80 of them against thousands of proxy ones, entirely because the only label control lives in a web page, and a web page is not where anyone is standing when it starts raining. The board has no RTC, so a power cut without a network gives a clock somewhere in 1970 on the next boot. Solar elevation, the diurnal harmonics and a sample's position on the 5-minute grid then all lie with complete confidence, and unlike a gap in the record the damage cannot be identified afterwards. train() now refuses a clock below 2025 or one that has stepped behind the newest stored row, and logs the refusal rather than training on fiction. Undervoltage and thermal capping both shift the SoC temperature, which is the regressor in the self-heating compensation, so a weak power supply presents as an unexplained temperature bias rather than as anything resembling a power problem. get_throttled is now sampled hourly and logged when set. Measured and deliberately not done: colour features. r, g and b are logged and 74% of rows carry usable colour, but adding blue/red, green/red and saturation made MAE 1.50% worse and helped in only 13 of 72 cases. Three more regressors on a 33-feature model whose longest horizon trains on 13 independent pairs is straightforwardly overfitting. That also prompted a sweep of the RLS prior and forgetting factor in both directions; delta = 100 with lambda = 0.9985 is a local optimum on both axes, so neither moved. |
||
|
kemal
|
f1647788c8 |
Learner hygiene: Hedge loss scale, look-ahead residual, conformal minimum
Three changes to ForecastHead and the conformal calibrator, each measured walk-forward on real data over seven train splits. Hedge normalised its losses by the current sample's worst loss, so on a quiet step where every member agreed to within 0.01 C whichever happened to be worst still took the full exp(-eta) penalty, exactly as if it had been wrong by 5 C. The regret bound assumes a fixed loss range, not a per-sample one, and the symptom was weights that jumped around with no relation to horizon. Normalising by the running member MAE instead is worth 1.81% of MAE, better on 106 of 126 heads, coverage unchanged. The residual handed to the conformal calibrator was computed after this sample's loss had already moved the weights, so it was better than anything the forecaster could produce and the intervals were calibrated about 2% too narrow. Coverage survived only because ACI notices the extra misses and reopens the band, a correction that should never have been needed. Scoring the blend with the pre-update weights leaves MAE untouched, as it must, and widens the intervals 2% to the honest width. The conformal quantile refused to produce a band below 20 scores. That number is arbitrary: the (1-alpha) empirical quantile is the ceil((k+1)(1-alpha))-th of k order statistics, so alpha = 0.10 needs 9. The 20 became actively harmful in the previous commit but one, because striding pairs by the horizon leaves a long-horizon head about 13 scores per refit. Twelve of eighteen heads therefore fell through to 1.645*sigma with sigma from an unconstrained x'Px, giving bands of +/- 45 C and +/- 115% relative humidity on a young station. Those cover, by being absurd, which is why the backtest never flagged them: a long walk-forward passes 20 scores early and never looks back. After the change all eighteen heads have a band from the first fit, +/- 3.1 C and +/- 7.7% in the same place. Also measured and deliberately not done: adding the Kalman level variance to the predictive spread. It moves sigma by 0.06% at the shortest horizon and 0.00% everywhere else, so the plumbing to carry it through three files buys nothing. |
||
|
kemal
|
41a46ae14a |
Take model tuning from config too, not from the saved state
The companion to the Kalman fix. RecursiveLeastSquares.from_dict and AdaptiveConformal.from_dict restore lambda, delta, alpha, gamma and the conformal window alongside their data, and load_dict replaces the config-built heads with those, so every one of those knobs was immutable on any station that already had state. Editing config.yaml and restarting looks exactly like a change with no effect, which is the failure mode that cost real time on the Kalman side of this before it was found. Only the estimate is state now. Weights, covariances and conformal scores are restored; tuning is re-applied from config on every load. The conformal deques are rebuilt when the configured window changes, preserving their contents. load_dict also skips heads for a target or horizon this build no longer has, rather than resurrecting them from a stale file. Found while implementing a damped-trend ensemble member, which was then abandoned: see the following note. |
||
|
kemal
|
3f41881c64 |
Take Kalman tuning from config, not from the saved state
q and r were written into station_state.json and restored over the configured values, so tuning was effectively immutable in the field. This was found the expensive way: the retune in the previous commit was deployed, the service restarted cleanly, and the filters carried on with q = 2e-6 because that is what the state file said. Measured median rate afterwards was 14.4 C/h against 12.4 before, which is to say nothing happened. Only the estimate is state. x, P and initialised are restored; q and r now come from config every time. P may be momentarily inconsistent with a changed q, which costs a few hundred samples of reconvergence and is far cheaper than a configuration change that appears to work and does not. load_dict also now skips filters this build no longer has, rather than resurrecting them from an older state file. |
||
|
kemal
|
40f934901d |
Retune the Kalman process noise, and fuse the two thermometers
Two changes to the same signal path, one large and one small. The large one: all three filters were tuned to track one to three decades faster than their signals move. In a still room the temperature filter reported a median rate of 12.4 C/h while the air moved 0.4 C/h, and it overshot a real -36 C/h event by 77%. Sweeping q against the RMSE of the reported rate versus the true rate, using noise measured on the board (temperature 0.088 C, pressure 0.022 hPa, humidity 0.40 %): temperature 6.45 -> 0.37 C/h RMSE 2e-6 -> 1e-9 pressure 2.15 -> 0.24 hPa/h RMSE 1e-5 -> 1e-8 humidity 27.94 -> 3.55 %/h RMSE 5e-5 -> 2e-8 Tracking does not suffer. Lag against a genuine 2 C/h ramp is 0.003 C at both the old and new values, and the peak response to a five-minute event moves closer to the truth rather than further from it, because the overshoot goes away. What is given up is response to sub-minute transients, which for a station forecasting fifteen minutes to a day ahead is noise to reject. This matters most for pressure, whose tendency drives the precipitation forecast, and which was the worst tuned of the three. config.yaml shadowed kalman_q_temp, so editing the dataclass alone changed nothing. All six values are now listed there with that hazard spelled out, because a silent shadow cost real time here. The small one: temp_raw was the plain average of two thermometers whose white-noise sds differ by 7x (LPS25HB 0.007 C, HTS221 0.049 C), which throws the quiet one away. Inverse-variance weighting cuts the raw noise 3.5x. The trap is that the chips do not agree. They sit at different distances from the SoC and stand about 1.3 C apart, so weighting by variance alone drags temp_raw 0.48 C onto the LPS25HB, which after the 1.55x gain of the inverse compensator is 0.75 C of silent bias on every reading, since k was fitted against the mean of the two. The gradient is therefore tracked and removed before weighting and only the deviations are fused: measured mean shift 0.0001 C, noise still 3.5x lower. The tracked gradient is retained because it is a second observation of self-heating. Also corrected: the earlier claim that the HTS221 was the quieter channel was wrong, taken from twelve samples at a cadence slow enough that real drift dominated. At 0.5 s over 120 samples the LPS25HB is quieter by 7x and takes 98% of the weight. |
||
|
kemal
|
e3176e29c9 |
Stride training pairs by the horizon instead of by the grid row
fit() trained every head on every consecutive grid row. At the 1 d horizon on
a 5-minute grid adjacent pairs share 287 of their 288 samples, so the filter
was handed the same outcome 288 times and RLS with forgetting read each one as
fresh evidence:
horizon steps overlap independent events in a 400-score window
15m 3 66.7% 133.3
1h 12 91.7% 33.3
3h 36 97.2% 11.1
6h 72 98.6% 5.6
12h 144 99.3% 2.8
1d 288 99.7% 1.4
The day-ahead head was therefore fitted on roughly two independent outcomes by
a filter carrying 667 updates of memory, and its interval was a 90th percentile
of a sample of size one.
This is not a compute shortcut that trades accuracy for speed. Measured
walk-forward on four days of real station data and averaged over five train
splits, striding improves every horizon past fifteen minutes:
15m +0.6% 1h -12.2% 3h -31.7% 6h -33.3% 12h -39.5% 1d -14.4%
with coverage unchanged at 87 to 92%, and the fit 11.6x faster. The redundancy
was not merely wasted work, it was collapsing P onto the one direction the
repeated sample excited.
The stride phase rotates each refit and is persisted, so a long-lived station
eventually trains on every offset rather than seeing one sample in 288 forever,
and a restart does not pin it to phase 0. A floor relaxes the stride when a
long horizon on a short record would otherwise yield one or two pairs; 12 was
chosen by sweeping it across five splits rather than picked.
Single-split runs showed 10 to 17% regressions at the 1 d horizon that moved
with the parameter. Averaging over five splits removed them, which is the
expected result for a head fitted and scored on under two independent
outcomes. That horizon cannot be evaluated on a four-day record and was not
tuned against.
Incidentally, this also retires the parallel-retrain idea: the Pi's 42 s
retrain becomes a few seconds, and multiprocessing inside a 280 MB cap buys
nothing for a job that short.
|
||
|
kemal
|
bda42a0468 |
Log both Sense HAT thermometers, and migrate schemas that predate them
The board carries two independent thermometers and the code averaged them into temp_raw without ever recording either. Measured over 12 samples on a real station: HTS221 30.973 C at sd 0.060, LPS25HB 29.810 C at sd 0.443, a standing gradient of 1.163 C with the SoC at 44.55 C. Two things follow from that and neither is possible without the raw channels. A plain average of a quiet sensor and one seven times noisier lands at sd 0.223 where inverse-variance weighting reaches 0.060, and the gradient between two chips at different distances from the SoC is a second observation of self-heating that could identify the compensator's k with no reference thermometer. Both need history, and history cannot be backfilled, so the columns land on their own ahead of the work that consumes them. CREATE TABLE IF NOT EXISTS is a no-op against a table that already exists, so adding to COLUMNS would have reached a fresh install and silently missed every station already running, then surfaced as an OperationalError inside insert_telemetry. That sits on the sample loop, so it takes a station down rather than leaving a gap. Store now reconciles the table against COLUMNS on open, which makes every future column addition safe rather than just this one. The simulator gains the same two channels, with couplings solved so their forward models average to exactly the k = 0.55 the compensator is tuned against. Aggregate behaviour is unchanged; only the per-channel detail is new. Simulated temp_raw noise does rise from 0.05 to 0.223, which is not a regression but the end of an over-optimistic figure: it was modelling the quiet sensor and calling it the average. |
||
|
kemal
|
485affe956 |
Fix the runaway forecasts: refits accumulated, and annual terms fitted too early
Reported from a real station after 1.5 days: a six hour temperature forecast of 53 C in a 24 C room, and 9 C at one day, both carrying a plus or minus of 0.43. Confidently wrong is the one failure this project is supposed to refuse. Root cause. fit() replayed history into the live RLS on every retrain tick and never reset, so 453 grid rows had produced 64,676 updates in a day and a half. RLS with forgetting reads every update as fresh evidence, so the model believed it had a hundred times the data it had: P collapsed, in-sample error looked excellent, and the weights drifted without bound in directions the data never excited. Measured: cond(P) 3.1e9 and ||theta|| 1680 against a median |theta| of 1.67. A refit now starts from the prior, which makes retraining idempotent. Across 25 refits on the real data ||theta|| holds at 11.35, drifting 0.03, where before it grew without limit. The two largest weights were sin_doy and cos_doy at +1174 and +1191. Annual harmonics were in the design matrix from the first sample, where they are near-constant, near-collinear with each other and with the bias, and a rank-deficient regressor is what RLS answers with enormous cancelling weights. They are now held at zero until the record spans the same 120 days the climatology fit already requires, because a day and a half of data says nothing whatsoever about the season. Also raised the standardiser's variance floor from 1e-8, which only caught a bit-exactly constant column, to 1e-3. A feature that merely barely moves was being divided by its own noise. The conformal calibrators and Hedge weights are deliberately not reset by a refit: those are earned from scored forecasts, not from this regression. Backtest unchanged within noise, coverage still 89 to 91 across all 18 heads. Four regression tests added, including that refitting the same history twice must give the same model. |
||
|
kemal
|
4cca40388f |
Heated environment: a thermostat member in the forecast ensemble
A room held at a setpoint is a different process from one left to drift. It is
a closed loop, and persistence, the baseline everything here is scored against,
is the wrong statement about it: the truth is not that it stays where it is, it
is that it returns to the setpoint.
So site.heating adds a fourth ensemble member, first order because that is what
a controlled system is:
dT_set(h) = (T_set - T_now) * (1 - exp(-h / tau))
Humidity follows and is the part that is easy to get wrong. Heating adds no
moisture, so vapour pressure is conserved and not relative humidity:
RH(h) = RH_now * es(T_now) / es(T_now + dT_set(h))
Warm the air and RH falls although nothing was dried, which is why a heated
house in winter is dry. The test asserts the dew point is unchanged to 1e-6.
Pressure gets zero: a thermostat cannot move the synoptic field.
Offered, not imposed. Hedge scores this member on realised error like any
other, so a wrong tau or a stale setpoint costs accuracy and gets down-weighted
rather than quietly biasing every forecast. Verified: on history with no
heating the ensemble assigned it weight 0.000. With heating off it returns zero
and is identical to persistence.
Going from three members to four means old saved heads must migrate.
from_dict reinitialises weights and member_mae. I missed member_mae first time
and it did not fail on load, it failed later inside learn() on a broadcast
error, which is a much worse place to find out; the migration test now covers
both and calls learn() to prove it.
Settings tab gains the toggle, setpoint and time constant. Turning heating on
or off is treated as a regime change like a door: discontinuity marker plus a
queued retrain.
|
||
|
kemal
|
db0f877052 |
Fix flaky row-count assertion in the recompute tests
Equality on the row count raced the live sample loop under TestClient, which legitimately inserts rows mid-test. Now asserts no rows are lost, which is the property that matters. Run three times to confirm it is stable. |
||
|
kemal
|
98210bff8f |
Six enhancements: recompute, markers, vendoring, tests, nerd stats, DS18B20
1. POST /api/recompute re-derives every compensated column from the untouched raw values, removing the step a calibration otherwise leaves through the history. Possible because temp_raw, cpu_temp and hum are never overwritten. Idempotent by construction and tested per row: 0 of 6051 rows change on a second run. 6069 rows in 0.25 s here, so a few seconds on the Pi. 2. Calibration now emits a 'discontinuity' event alongside the calibration log, so downstream views can find the boundary without parsing prose. 3. Vendored Tailwind, Chart.js, hammer, the zoom plugin, KaTeX with its 20 woff2 faces, and both Google fonts into ashvale/static, served by the station. 1.4 MB. Verified with every non-localhost request aborted in the browser: zero external requests, equations still render, fonts still load. The dashboard no longer needs internet. 4. 54 pytest cases over the pure numerics: physics closed forms and round trips, both compensator inverse properties, the Kalman covariance invariants and NIS consistency, the RLS trace cap under a deliberately unexcited regressor, conformal coverage, and the Zambretti ordering. Wired into CI after the seed step so the recompute cases have history. Writing them caught my own sign error on the conformal update: a hit raises alpha and narrows the band, which reads backwards until you follow it through. 5. Stats for Nerds gains the condition number of each head's covariance, a standardised innovation histogram per Kalman filter from a bounded 600 sample ring buffer, and a reliability strip of realised against nominal coverage. All arithmetic on data already in memory. 6. OutdoorProbe reads a DS18B20 over the kernel 1-Wire driver, no new dependency. Polled on its own slower cadence because the sensor blocks for up to 750 ms during conversion, which would eat a third of the 2 s sample budget. Rejects the 85000 power-on sentinel and out-of-range values, and reports age so a dead probe cannot masquerade as fresh. |