Two changes to the same signal path, one large and one small.
The large one: all three filters were tuned to track one to three decades
faster than their signals move. In a still room the temperature filter
reported a median rate of 12.4 C/h while the air moved 0.4 C/h, and it
overshot a real -36 C/h event by 77%. Sweeping q against the RMSE of the
reported rate versus the true rate, using noise measured on the board
(temperature 0.088 C, pressure 0.022 hPa, humidity 0.40 %):
temperature 6.45 -> 0.37 C/h RMSE 2e-6 -> 1e-9
pressure 2.15 -> 0.24 hPa/h RMSE 1e-5 -> 1e-8
humidity 27.94 -> 3.55 %/h RMSE 5e-5 -> 2e-8
Tracking does not suffer. Lag against a genuine 2 C/h ramp is 0.003 C at both
the old and new values, and the peak response to a five-minute event moves
closer to the truth rather than further from it, because the overshoot goes
away. What is given up is response to sub-minute transients, which for a
station forecasting fifteen minutes to a day ahead is noise to reject.
This matters most for pressure, whose tendency drives the precipitation
forecast, and which was the worst tuned of the three.
config.yaml shadowed kalman_q_temp, so editing the dataclass alone changed
nothing. All six values are now listed there with that hazard spelled out,
because a silent shadow cost real time here.
The small one: temp_raw was the plain average of two thermometers whose
white-noise sds differ by 7x (LPS25HB 0.007 C, HTS221 0.049 C), which throws
the quiet one away. Inverse-variance weighting cuts the raw noise 3.5x.
The trap is that the chips do not agree. They sit at different distances from
the SoC and stand about 1.3 C apart, so weighting by variance alone drags
temp_raw 0.48 C onto the LPS25HB, which after the 1.55x gain of the inverse
compensator is 0.75 C of silent bias on every reading, since k was fitted
against the mean of the two. The gradient is therefore tracked and removed
before weighting and only the deviations are fused: measured mean shift
0.0001 C, noise still 3.5x lower. The tracked gradient is retained because it
is a second observation of self-heating.
Also corrected: the earlier claim that the HTS221 was the quieter channel was
wrong, taken from twelve samples at a cadence slow enough that real drift
dominated. At 0.5 s over 120 samples the LPS25HB is quieter by 7x and takes
98% of the weight.
Co-Authored-By: Claude Opus 5 <[email protected]>
fit() trained every head on every consecutive grid row. At the 1 d horizon on
a 5-minute grid adjacent pairs share 287 of their 288 samples, so the filter
was handed the same outcome 288 times and RLS with forgetting read each one as
fresh evidence:
horizon steps overlap independent events in a 400-score window
15m 3 66.7% 133.3
1h 12 91.7% 33.3
3h 36 97.2% 11.1
6h 72 98.6% 5.6
12h 144 99.3% 2.8
1d 288 99.7% 1.4
The day-ahead head was therefore fitted on roughly two independent outcomes by
a filter carrying 667 updates of memory, and its interval was a 90th percentile
of a sample of size one.
This is not a compute shortcut that trades accuracy for speed. Measured
walk-forward on four days of real station data and averaged over five train
splits, striding improves every horizon past fifteen minutes:
15m +0.6% 1h -12.2% 3h -31.7% 6h -33.3% 12h -39.5% 1d -14.4%
with coverage unchanged at 87 to 92%, and the fit 11.6x faster. The redundancy
was not merely wasted work, it was collapsing P onto the one direction the
repeated sample excited.
The stride phase rotates each refit and is persisted, so a long-lived station
eventually trains on every offset rather than seeing one sample in 288 forever,
and a restart does not pin it to phase 0. A floor relaxes the stride when a
long horizon on a short record would otherwise yield one or two pairs; 12 was
chosen by sweeping it across five splits rather than picked.
Single-split runs showed 10 to 17% regressions at the 1 d horizon that moved
with the parameter. Averaging over five splits removed them, which is the
expected result for a head fitted and scored on under two independent
outcomes. That horizon cannot be evaluated on a four-day record and was not
tuned against.
Incidentally, this also retires the parallel-retrain idea: the Pi's 42 s
retrain becomes a few seconds, and multiprocessing inside a 280 MB cap buys
nothing for a job that short.
Co-Authored-By: Claude Opus 5 <[email protected]>
A room held at a setpoint is a different process from one left to drift. It is
a closed loop, and persistence, the baseline everything here is scored against,
is the wrong statement about it: the truth is not that it stays where it is, it
is that it returns to the setpoint.
So site.heating adds a fourth ensemble member, first order because that is what
a controlled system is:
dT_set(h) = (T_set - T_now) * (1 - exp(-h / tau))
Humidity follows and is the part that is easy to get wrong. Heating adds no
moisture, so vapour pressure is conserved and not relative humidity:
RH(h) = RH_now * es(T_now) / es(T_now + dT_set(h))
Warm the air and RH falls although nothing was dried, which is why a heated
house in winter is dry. The test asserts the dew point is unchanged to 1e-6.
Pressure gets zero: a thermostat cannot move the synoptic field.
Offered, not imposed. Hedge scores this member on realised error like any
other, so a wrong tau or a stale setpoint costs accuracy and gets down-weighted
rather than quietly biasing every forecast. Verified: on history with no
heating the ensemble assigned it weight 0.000. With heating off it returns zero
and is identical to persistence.
Going from three members to four means old saved heads must migrate.
from_dict reinitialises weights and member_mae. I missed member_mae first time
and it did not fail on load, it failed later inside learn() on a broadcast
error, which is a much worse place to find out; the migration test now covers
both and calls learn() to prove it.
Settings tab gains the toggle, setpoint and time constant. Turning heating on
or off is treated as a regime change like a door: discontinuity marker plus a
queued retrain.
Everything that was previously a curl command now has a surface: surroundings,
site geometry, the matrix, the psychrometric flag, and the maintenance actions.
Changes persist to data/state/settings.json, not config.yaml. That file is
hand-annotated and hand-edited per station, and rewriting it from an API would
destroy the comments and risk clobbering something the owner set. The overlay is
applied last in load_config, so a dashboard change beats both the file and the
environment, and deleting the overlay reverts everything. Written atomically via
a temp file so a crash cannot truncate it.
Every field applies live. A settings page that needs a restart is one people
stop trusting, so site geometry is re-read per sample, the compensator flag is
set on the live object, and the display picks up its rate the next frame.
Two deliberate frictions. Selecting a surroundings pill only stages it: nothing
is recorded until you press the button, because that writes a discontinuity
marker and queues a retrain. And changing altitude or the psychrometric flag
says outright that the stored history is now inconsistent and offers the
re-derive, rather than leaving a silent mismatch.
Verified in a browser: pills stage and apply, the toggle round-trips, re-derive
ran 6201 rows in 0.29 s from the button, all six tabs report zero scrollbars and
zero clipping, zero console errors.
recompute replayed the Kalman over stored rows at their own spacing while q
stays tuned for the live 2 s cadence. Q scales with dt^3, so at the 30 s
persist interval the process noise was 3375x too large and the filter tracked
noise instead of smoothing: it wrote indoor temperature rates of +/-20 C/h into
the history. This is the exact trap DESIGN.md section 2 documents for
simulate.py, which does scale q, and I walked into it anyway. Now rescaled per
step, because tiering means the stored cadence is not constant. Mean |rate| on
the real board dropped to 2.73 C/h; what remains above 10 is the filter's
warm-up transient in the first four samples, which is honest.
Readout scene puts the actual numbers between the animations: temperature,
humidity, sea-level pressure and the signed three hour forecast, each in its
channel colour, scrolling. Text is drawn whole-pixel on purpose. Everything
else here is sub-pixel and that is what makes it look good, but splitting a
3 px glyph across two columns halves its peak and smears it illegible. Crisp
beats smooth when the thing has to be read.
site.environment and site.enclosure record where the sensor lives and what has
changed around it, with POST /api/environment to change them at runtime. This
is not cosmetic: closing a door changes how strongly the sensor couples to
outside, which is a regime change in the process the heads are fitting, and at
lambda 0.9985 they carry about 55 hours of memory. Left alone they keep
predicting the old room for two days. Page-Hinkley would notice eventually but
needs matured forecasts to do it, which at the long horizons is the same two
days. So the endpoint marks a discontinuity and queues a retrain.
24 fps costs about 11% of one core on a Zero 2 W, measured on the board. That
is a reasonable default for something you look at, but it is a decorative load
sharing a 512 MB machine with the forecaster, so it should be the owner's
choice. server.led_fps is clamped to 4..30.
Particle fall speed now divides by the configured rate rather than the module
constant, so rain falls at the same real-world speed whatever the frame rate,
instead of slowing down when you turn the frame rate down.
1. POST /api/recompute re-derives every compensated column from the untouched
raw values, removing the step a calibration otherwise leaves through the
history. Possible because temp_raw, cpu_temp and hum are never overwritten.
Idempotent by construction and tested per row: 0 of 6051 rows change on a
second run. 6069 rows in 0.25 s here, so a few seconds on the Pi.
2. Calibration now emits a 'discontinuity' event alongside the calibration log,
so downstream views can find the boundary without parsing prose.
3. Vendored Tailwind, Chart.js, hammer, the zoom plugin, KaTeX with its 20
woff2 faces, and both Google fonts into ashvale/static, served by the
station. 1.4 MB. Verified with every non-localhost request aborted in the
browser: zero external requests, equations still render, fonts still load.
The dashboard no longer needs internet.
4. 54 pytest cases over the pure numerics: physics closed forms and round
trips, both compensator inverse properties, the Kalman covariance
invariants and NIS consistency, the RLS trace cap under a deliberately
unexcited regressor, conformal coverage, and the Zambretti ordering. Wired
into CI after the seed step so the recompute cases have history. Writing
them caught my own sign error on the conformal update: a hit raises alpha
and narrows the band, which reads backwards until you follow it through.
5. Stats for Nerds gains the condition number of each head's covariance, a
standardised innovation histogram per Kalman filter from a bounded 600
sample ring buffer, and a reliability strip of realised against nominal
coverage. All arithmetic on data already in memory.
6. OutdoorProbe reads a DS18B20 over the kernel 1-Wire driver, no new
dependency. Polled on its own slower cadence because the sensor blocks for
up to 750 ms during conversion, which would eat a third of the 2 s sample
budget. Rejects the 85000 power-on sentinel and out-of-range values, and
reports age so a dead probe cannot masquerade as fresh.
Seven day outlook moves from History to Live, which now runs four rows.
Conditions ahead tightened so the Live column no longer needs a scroller.
Adds HumidityCompensator: an additive RH offset estimated by one-step RLS from
a trusted hygrometer, clamped to +/-35%, persisted, exposed at
POST /api/calibrate/humidity and on the renamed Models and calibration tab.
It also implements the psychrometric term (RH moved from element temperature
onto air temperature via conserved vapour pressure) but leaves it OFF by
default. The thermal argument predicts a hot element reads low; measured
against a reference hygrometer this board read 75.4% where the truth was
50.4%, so it reads HIGH and that correction would push it the wrong way. When
the flag is enabled, simulate.py applies the exact inverse, per the
simulator/compensator trap in DESIGN.md section 2.
Models pane rebuilt: the scorecard is one column per target so all 18 heads
are visible, and no panel on the tab uses an internal scroller. Verified in
Chromium at 1600x900: Live, History and Models all report zero scrollbars,
zero clipping, no page scroll, zero console errors. Backtest is numerically
identical to the previous commit, confirming the humidity work is a no-op
while the flag is off.
kemalandClaude Opus 5