ashvale-station/docs/DESIGN.md

# Design notes

Internals, tuning and failure modes. This document deliberately does **not**
repeat the README: no pitch, no install instructions, no feature list, no API
table. It covers what you need when changing the code or debugging a station
that is behaving oddly, and nothing else.

Read this before touching anything under `ashvale/models/`.

---

## 1. The data model

### Schema

One wide table, `telemetry`, keyed on a float epoch `ts`. Every column is
`REAL` except `tier`. There is no normalisation, because a sensor sample is a
single denormalised event and joins on a Pi are not free.

| Column | Meaning |
|---|---|
| `temp_raw` | Straight off the HTS221/LPS25HB average, uncompensated |
| `temp_c` | After self-heating compensation, before filtering |
| `temp_smooth`, `temp_rate` | Kalman level and rate (°C, °C/h) |
| `hum`, `hum_smooth` | Raw and filtered relative humidity |
| `press`, `press_slp` | Station pressure and its sea-level reduction |
| `press_smooth`, `press_rate` | Kalman level and rate (hPa, hPa/h) |
| `cpu_temp` | SoC thermal zone. The nuisance variable |
| `dew_c` | Magnus dew point from smoothed inputs |
| `lux`, `r`, `g`, `b` | TCS3400 clear and colour channels |
| `pitch` … `gz` | IMU, nine values |
| `tier` | 0 raw, 1 five-minute mean, 2 hourly mean |

Supporting tables: `forecasts` (issued, valid, target, mu, lo, hi), `labels`
(human ground truth), `scores` (verification output), `events` (station log).

**Three columns are named `r`, `g` and `b`.** This has already caused one bug:
`GROUP BY b` in an aggregation query silently grouped by the blue channel,
because SQLite resolves an unqualified identifier to a real column in
preference to a result alias. It returned one row per sample while cheerfully
reporting the requested bucket size. Never use a single-letter alias in this
schema.

### Tiering

Nothing is deleted, only downsampled. `Store.compact()` folds raw rows older
than `raw_retention_days` into five-minute means, and five-minute rows older
than `five_min_retention_days` into hourly means. A year of history lands
around 30 MB.

The `tier` column exists so you can tell a genuine hourly observation from a
mean of twelve. Range queries deliberately mix tiers, which is right for
display but means **a query spanning the raw/five-minute boundary has
non-uniform effective resolution**. If you ever compute a statistic that
assumes equal weight per row, weight by tier or restrict to one.

### Why aggregation happens in SQL

`range_series()` buckets with `CAST(ts/bucket AS INTEGER)*bucket` and
`AVG`/`MIN`/`MAX` inside SQLite. Pulling ninety days of rows into numpy to
average them costs more resident memory than the board has. The bucket is
chosen from the span and snapped to a ladder of familiar durations, targeting
about 700 points, because neither a browser nor a human benefits from more.

Min and max travel alongside the mean so the UI can shade a true range band. A
two-hour bucket that spanned four degrees should not render as a flat line.

---

## 2. The estimation layer

### Self-heating compensation

Model: `T = T_raw − k·(T_cpu − T_raw)`, with `k ≥ 0` estimated by recursive
least squares against any trusted reference you supply.

The regressor is `φ = max(T_cpu − T_raw, 0)` and the target is
`T_raw − T_ref`, so `k·φ` should equal the observed bias. One step, forgetting
factor 0.98:

```
gain = P·φ / (λ + φ·P·φ)
k   ← clip(k + gain·(target − k·φ), k_min, k_max)
P   ← (P − gain·φ·P) / λ
```

Measured behaviour: recovers a true `k = 0.62` from a prior of `0.30` in a
**single sample**, and holds post-calibration bias to 0.012 °C over 200
subsequent readings.

**The clamp is not decoration.** A single mistyped reference drives `k` to its
bound, and because state persists across restarts it stays there, quietly
biasing every reading until you notice. `POST /api/calibrate {"reset": true}`
exists for exactly that.

**If you write a simulator or a test fixture, the forward model must be the
exact inverse:** `T_raw = (T + k·T_cpu)/(1 + k)`. Generating the bias as
`T + k·(T_cpu − T)` is a different relation, and the mismatch injects roughly
1.2 °C of phantom noise floor that caps every skill score. This has already
happened once.

### The Kalman bank

One constant-velocity filter per signal. State `x = [level, rate]`, standard
continuous white-noise-acceleration process model:

```
F = [[1, Δt], [0, 1]]
Q = q · [[Δt³/3, Δt²/2], [Δt²/2, Δt]]
H = [1, 0]
```

Two implementation points that matter.

**Joseph form.** The update is `P ← (I−KH)·P·(I−KH)ᵀ + K·R·Kᵀ`, not the
shorter `P ← (I−KH)·P`. The short form accumulates asymmetry and loses positive
semi-definiteness over months of continuous running, and nobody notices until
the filter quietly stops working.

**`q` is tuned for the live 2 s cadence.** Because `Q` scales with `Δt³`,
running the same filter at a 300 s step makes the process noise five orders of
magnitude larger, at which point the filter abandons smoothing and tracks
measurement noise. `scripts/simulate.py` therefore scales `q` by
`(sample_period/step)³` when backfilling. Before that fix, rate estimates blew
past anything physical and enshrined a −37 hPa/h all-time record.

`nis` (normalised innovation squared) is exposed per filter. It should hover
near 1. Persistently high means the filter is too confident and lagging real
change; persistently low means you are over-smoothing.

---

## 3. Features

33 columns, defined once in `FEATURE_NAMES`. An assertion in
`build_features()` fails loudly if the stacked matrix width drifts from that
list, which is the cheapest guard available against a silently misaligned
design matrix.

Three rules govern what goes in.

1. **Anything closed-form is computed, not learned.** Dew point, wet bulb,
   VPD, absolute humidity, solar elevation and the clear-sky cloud index come
   from `physics.py`. Making a linear learner rediscover the Magnus curve from
   data wastes both samples and capacity.
2. **Anything periodic is a sine/cosine pair.** Two diurnal harmonics and one
   annual, so phase is representable without a discontinuity at midnight.
3. **Lags are expressed in hours, not samples.** `press_tend_3h` means three
   hours whatever `grid_s` is. Changing the grid must not silently change what
   the model means by "three hours ago".

### Standardisation

`Standardiser` keeps streaming Welford moments and z-scores everything except
the bias column. Not optional: unscaled pressure sits near 1013 while unscaled
temperature rate sits near 0.02, and the resulting condition number will
embarrass you.

### Solar geometry as a feature

`solar_position()` is the NOAA low-precision model, accurate to a few tenths of
a degree and costing about twenty floating point operations. It gives the
diurnal cycle real physical structure rather than making the model infer it
from clock time alone. `clear_sky_irradiance()` turns measured lux into a crude
cloudiness index by ratio. Through a south-facing window that is a surprisingly
decent okta estimate; in a north-facing room it is nearly useless. Judge
accordingly.

---

## 4. The forecasting core

### Why RLS and not gradient descent

A station produces 288 grid rows a day. Sample efficiency dominates everything
else. RLS is the exact minimiser of the exponentially weighted squared error at
every step, not an approximation, so it converges in far fewer samples than
SGD. The covariance `P` is a genuine parameter-uncertainty estimate, free. One
`(33, 33)` matrix is about 8 kB, so the whole bank of 18 fits in L2 cache on a
Cortex-A53.

### The covariance trace cap

```
P ← (P − outer(gain, P·x)) / λ
P ← (P + Pᵀ)/2                    # enforce symmetry
if trace(P) > p_max: P *= p_max/trace(P)
```

**This is the single most important guard in the file.** Plain exponential
forgetting inflates `P` without bound along directions the data never excites.
On a quiet night the regressor barely moves, `P` grows exponentially in the
unexcited subspace, and the model detonates on the first sunrise sample. It is
the most common way a field RLS deployment dies. If you refactor `rls.py`, keep
both the cap and the symmetrisation.

### Direct heads, not recursion

18 heads: 3 targets × 6 horizons. Each predicts a **delta from now**, and the
absolute forecast is reconstructed as `anchor + delta`.

Predicting deltas rather than levels matters more than it looks. A model that
must output 14.7 °C spends its capacity representing the mean; one that outputs
+0.4 °C spends it on the weather.

Iterating a single one-step model 288 times to reach 24 hours would compound
its own bias into a beautifully smooth lie. 18 direct heads cost about 150 kB
total and each is honest about its own horizon.

### The Hedge ensemble

Each head blends three members: persistence (delta = 0), climatology (delta
from the harmonic fit) and the learned RLS output. Weights update by
exponentiated gradient on normalised absolute loss, then renormalise. This
guarantees the ensemble is never much worse than its best member, and it
re-weights within about a day when the season turns.

**The ensemble is allowed to conclude the learned model is useless.** At
15-minute pressure it typically parks most of its weight on persistence. That
is correct behaviour surfaced honestly, not a defect to engineer away.

### Adaptive conformal intervals

Split conformal is valid only under exchangeability, and weather is emphatically
not exchangeable: a front arrives and yesterday's residual quantile becomes
fiction. Adaptive conformal inference (Gibbs and Candès) feeds realised coverage
back into the working α:

```
α ← clip(α + γ·(α_target − 1[y ∈ C]), 0.005, 0.75)
```

The band widens after each miss and narrows after each hit, so long-run
coverage tracks the target whatever the distribution does underneath.

**Coverage is the acceptance test for any change to this path.** Measured
coverage sits at 89 to 91% against a 90% target across all 18 heads. A change
that improves MAE while coverage drifts to 70% is a regression, not an
improvement, because the intervals have started lying.

---

## 5. Verification

`Station.verify()` runs every five minutes. It pulls forecasts whose validity
time has passed, looks up the truth and the anchor, and computes per bucket:

```
skill = 1 − MAE_model / MAE_persistence
```

Zero means no better than assuming nothing changes. **Negative is useful
information, not an embarrassment**: it says ship persistence at that horizon
and stop pretending.

Scored errors feed two places: the conformal calibrator for each head, and, for
horizons up to 3 h, the Page-Hinkley drift detector. Forecasts more than an
hour past validity are deleted so the table does not grow without bound.

A forecast that is never scored is an opinion. A forecast scored against
persistence is a measurement.

### Reproducible evidence

`scripts/simulate.py` takes `--seed` and `--end`. **Both must be pinned** for
run-to-run comparability. The seed fixes the OU realisation, but the wall-clock
anchor moves solar elevation and the seasonal harmonic, so an unpinned `--end`
changes temperature and humidity while leaving pressure bit-identical. That
asymmetry is a useful diagnostic in itself: if pressure differs between two runs
with the same seed, something other than the anchor has changed.

Note the interaction with `evaluate.py`: `store.window()` looks back from *now*
with a 60-day default, so a history pinned to a date in the past reports "not
enough history" unless you pass a wide `--hours`.

---

## 6. Monitoring

Three detectors, because they fail differently.

**`MahalanobisEWMA`** catches abrupt multivariate novelty: a window opening, a
heater cycling, a squall. The EWMA on the whitened residual gives
persistence-aware detection, so one odd sample is noise and ten in a row is an
event. Shrinkage toward a scaled identity is **not optional**: with six signals
the sample covariance is singular for the first hour, and a singular covariance
turns Mahalanobis distance into a random number generator with an authoritative
name.

**`PageHinkley`** catches slow change: a sensor drifting, a season turning, a
model going stale. It runs on forecast error and **triggers retraining**, which
is a far better signal than a cron schedule.

**`SensorHealth`** catches the quietest failure of all. A latched sensor looks
perfectly normal to both detectors above: the dashboard is fine, the model
trains happily, and every forecast is confidently wrong. Bit-identical
consecutive readings are the only tell.

---

## 7. Tuning

| Symptom | Knob | Direction |
|---|---|---|
| Temperature reads consistently high | Calibrate from the Models tab, or `sensor.cpu_heat_k` | Raise |
| Readings over-smoothed, lag real change | `sensor.kalman_q_temp` | Raise |
| Rates look noisy | `sensor.kalman_q_*` down, or `kalman_r_*` up | |
| NIS persistently much above 1 | Filter too confident, raise `q` | Raise |
| NIS persistently much below 1 | Over-smoothing, lower `q` | Lower |
| Adapts too slowly to a season change | `model.rls_forgetting` toward 0.995 | Lower |
| Jumpy, forgets overnight | `model.rls_forgetting` toward 0.9995 | Raise |
| Coverage well below target | `model.conformal_gamma` | Raise |
| Coverage well above target, lazily wide bands | `model.conformal_gamma` | Lower |
| Drift alarms constantly | `model.drift_lambda` | Raise |
| Retrains eat the CPU | `model.train_period_s` up, `max_pairs` down | |
| Zambretti reads pessimistic everywhere | `site.altitude_m` is wrong | Fix it |

On forgetting factors: effective memory is `1/(1−λ)` samples. At `λ = 0.9985`
on a five-minute grid that is about 667 samples, roughly 55 hours. `λ = 0.99`
is about two hours and will chase noise.

---

## 8. Performance budget on a Zero 2 W

Roughly 20× slower than a modern x86 core. Measured:

| Operation | x86 | Zero 2 W |
|---|---|---|
| Backfill 4032 rows | 0.6 s | 11.8 s |
| Full retrain, 18 heads | ~5 s | 60 to 100 s |
| Resident set, steady state | | ~150 MB |
| Database, one year | | ~30 MB |

Retraining runs in a worker thread via `asyncio.to_thread`, so the sample loop,
the API and the LED never stall. A command that looks hung on the Pi is usually
just the Pi.

`max_pairs` in `NowcastEnsemble.fit` caps supervised pairs per head at the most
recent 2500. Not a shortcut: with `λ = 0.9985` the 4000th-most-recent sample
carries a weight of about `e⁻⁶`. It costs real seconds on a Cortex-A53 and buys
nothing measurable.

**One uvicorn worker, deliberately.** The station owns mutable model state; a
second worker would give you two divergent forecasters sharing a socket.

---

## 9. Extension points

**Adding a target.** Append to `model.targets` in config and ensure
`station.py` resamples it into the `series` dict; the head bank scales
automatically. Add a climatology column too, or the ensemble's climatology
member returns zero delta forever and quietly wastes a third of its weight.

**Adding a feature.** Append to `FEATURE_NAMES` **and** to the `column_stack`
in `build_features()`, in the same position. The width assertion catches
mismatches. Persisted RLS state from before the change is now the wrong
dimension: delete `data/state/station_state.json` and retrain.

**Adding a sensor.** `sensors.py` is the only module that touches hardware.
Extend `SenseBoard.read()`, add the column to `storage.COLUMNS` and the schema,
and mirror it in `SimulatedBoard` so the simulator still exercises every path.
A DS18B20 outside the window is the highest-value hardware change available: it
removes the indoor caveat entirely and improves every model at once.

**Replacing the learner.** `ForecastHead` needs only `predict(x)` and
`update(x, y)`. Anything with that interface drops in. If you swap in something
without a covariance you lose `predict_std`, and the conformal fallback `sigma`
becomes meaningless until enough residuals accumulate.

---

## 10. Things that will bite you

- **Station pressure passed where sea-level pressure is expected.** Silent, and
  at 100 m elevation it shifts the Zambretti number by about two categories,
  permanently.
- **Single-letter SQL aliases.** See section 1.
- **`$` as a JavaScript identifier** in `dashboard.py`. It collides with
  bundled libraries and kills the entire script with one opaque syntax error.
  The helper is `el()`.
- **Unconstrained Chart.js canvases.** With `maintainAspectRatio: false` a
  chart expands to fill its parent. Every canvas needs an explicitly sized
  relative wrapper or it swallows its panel.
- **Charts inside `display: none`.** Chart.js cannot measure a hidden canvas,
  so tab switches call `resize()` on reveal.
- **Persisted state outliving a schema change.** `station_state.json` holds
  trained parameters with fixed dimensions. Change the feature count without
  deleting it and you get a shape error at the first update, or worse, silence.
- **Synthetic history left in the database.** After a `simulate.py` backfill,
  clear the synthetic rows once real telemetry accumulates, or the model stays
  anchored on a stochastic weather model rather than on your room.