amorce · an outsider's lab on Timestral · J1r

Real turbines

Used zero-shot as a normal-behaviour model on the temperatures of real turbines, how many of the 45 documented faults of CARE to Compare does Timestral see before they happen, how early, at the same false-alarm rate as a ridge regression?

verified by a fresh sessionseeds 0

What we expected (written before the run)

Each expectation was written with the hypotheses, before any number existed; next to it, the status of the hypotheses it rests on, from the results: 6 of 7 held.

A matrix of 45 rows (one per fault, grouped by farm, with the operator's description) by five columns (four Timestral sizes, then the ridge); green with the lead time in days when the turbine alarmed inside the event window before the fault, grey when missed; a click shows the temperature that alarmed first, the column's 80 % band, the event start, the first alarm and the fault. held · H1r.4 held · H1r.5 held · H1r.7
Under it, per column, the CUSUM limit h found by replay, the false alarms per turbine-year on the calibration periods (about 4, the target) and on the 50 normal datasets, the faults detected next to what chance alone would detect, and the paired-bootstrap interval of the difference with the ridge. refuted · H1r.1 held · H1r.2 held · H1r.3
A PIT histogram of the calibration periods, one line per column, against the flat line of a calibrated model. held · H1r.6

Figures

The 45 faults · one row per anomaly event, one column per model

Green = an alarm of the turbine (any monitored temperature) inside the event window, before the fault; the cell gives the lead time in days before the fault. Grey = missed. Every column on its own CUSUM limit, set by replay on the calibration periods for 4 false alarms per turbine-year. Click a cell: the temperature that alarmed first (or the first monitored one), the column's 80 % band, the event start, the first alarm and the fault.

Per column · threshold, false alarms, faults detected, and the paired bootstrap against the ridge

h: the CUSUM limit found by replay. Bootstrap: 2000 resamples of the datasets, the same draws for both columns; the interval is 95 %. A column beats the ridge only when the whole interval of the difference is above 0.

Faults detected out of 45 · and what chance alone would detect at the same false-alarm rate

Chance: the expected number of event windows an alarm process with the column's false-alarm rate on the normal datasets would hit, Σ 1 − exp(−rate × window).

False alarms per turbine-year · on the calibration periods (the target is 4) and on the 50 normal datasets

The calibration rate is where replay put each column; the normal datasets say whether it held on new weeks.

PIT histogram on the calibration periods · where the truth falls, every monitored temperature

Ten bins; a calibrated model is flat at 0.10. Mass at the ends: bands too narrow; mass on one side: a biased median.

What is watched · one temperature per component family and farm, and the four conditions

Chosen from each farm's feature description before any series was read. Farm A's status comes from the operator's fault logbook: it filters the ridge's training rows only, and every hour of farm A is scored.

Hypotheses, written before the run

id	claim	measured	status
`H1r.1`	The thresholds set by replay hold on new weeks - on the 50 normal datasets, every column stays at or under 8 false alarms per turbine-year, twice the target. On the 50 normal datasets the worst column raised 36.1 false alarms per turbine-year, over twice the target of 4 met on the calibration periods: the replayed threshold did not hold on new weeks (5M 11.9, 28M 10.1, 123M 11.6, 209M 11.2, ridge 36.1).	36.14	refuted · metrics.fa_per_year_normal.max_over_columns ≤ 8
`H1r.2`	As on the trap bench, no Timestral size detects more faults than the ridge beyond the paired-bootstrap dispersion.	0	held · metrics.sizes_beating_ridge = 0
`H1r.3`	Size buys little - the 209M detects at most 3 more of the 45 faults than the 5M.	2	held · metrics.detected_gain_209M_vs_5M ≤ 3
`H1r.4`	No column detects more than half of the 45 faults (at most 22), because many of them (pitch, communication, wiring, converter fuses) have no monitored temperature.	22	held · metrics.detected.max_over_columns ≤ 22
`H1r.5`	On farm B, where every fault sits on a component whose temperature is watched (three transformer cells, three rotor bearings), the 5M detects at least 4 of the 6.	6	held · metrics.detected_by_farm.B.Timestral-5M ≥ 4
`H1r.6`	On the calibration periods, the PIT of Timestral-5M averages between 0.40 and 0.60 over every monitored temperature, so its bands are centred on real temperatures as they were on electricity.	0.5013	held · metrics.pit_mean_cal.Timestral-5M ∈ [0.4, 0.6]
`H1r.7`	When the 5M detects a fault, it does so a median of at least 2 days before it happens.	15.04	held · metrics.lead_days_median.Timestral-5M ≥ 2

What the page asserts (checked at build time)

claim	value	check
All 95 datasets were scored, each on every monitored temperature with enough calibration days.	95	metrics.n_datasets = 95
Every column's CUSUM limit was found by replay and met the same target of 4 false alarms per turbine-year on the calibration periods.	1	metrics.calibration_reached_all = 1
On the calibration periods Timestral's 21 quantiles are calibrated on real temperatures: every size's mean PIT is within 0.01 of 0.5.	0.0042	metrics.pit_mean_cal_max_dev_sizes ≤ 0.01
No Timestral size detects more than 2 faults beyond what chance would detect at its own false-alarm rate.	1.66	metrics.detected_beyond_chance.max_over_sizes ≤ 2
On the normal datasets the ridge raises at least twice as many false alarms per turbine-year as any Timestral size.	3.03	metrics.fa_ratio_ridge_over_sizes_max ≥ 2
On the normal datasets every column raises more than twice its calibration rate of false alarms.	3.35	metrics.fa_drift_min_over_columns > 2

What surprised us

Detected, but not beyond chance. Each column hits about as many event windows as an alarm process with its own false-alarm rate would hit by luck: 5M 14 against 16.7, 28M 16 against 14.9, 123M 18 against 16.3, 209M 16 against 16.0, ridge 22 against 29.7. On these 45 faults, a daily temperature score with a replayed threshold does not separate a fault from its own background, zero-shot or ridge; the detection hypotheses that hold (H1r.2, H1r.3, H1r.4, H1r.5, H1r.7) say little.
The replayed threshold did not hold on new weeks. On the calibration periods every column sat near the target (2.8 to 3.6 false alarms per turbine-year for Timestral, 3.8 for the ridge); on the 50 normal datasets, 10.1 to 11.9 for Timestral and 36.1 for the ridge. Timestral's rolling context follows the turbine; the ridge, fitted on hours that end before the calibration period and the gap, does not.
Timestral's quantiles are calibrated on real temperatures: on the calibration periods its PIT histogram is flat and every size's mean PIT is within 0.004 of 0.5 (the ridge: 0.508). The forecast is good; the alarm built on it is what fails, as on the trap bench.
Farm B, where every fault sits on a watched component, is where every column scores (5M 6, 28M 5, 123M 5, 209M 5, ridge 6 of 6); its event windows are also the longest (median 36.5 days, against 7 on farm A and 15 on farm C), and chance alone would hit 4.1, 3.8, 4.0, 4.0, 5.7 of them.
Size changes little: 14 to 18 detections across the four sizes, the ridge 22. The paired bootstrap finds no size above the ridge in detections (5M below it), and every size below it in false alarms per turbine-year on the normal datasets (5M -24.2 [-34.8, -13.4], 28M -26.1 [-37.1, -14.9], 123M -24.6 [-35.0, -14.4], 209M -25.0 [-35.5, -14.8]).

Reproduce

make reproduce J=1r
# $(PY) -m amorce.care --download
# $(PY) -m amorce.jalons.j1r --out results/J1r.json

`make reproduce J=1r` fetches CARE to Compare from Zenodo when it is absent (5.5 GB, checked against the md5 Zenodo publishes), reads it member by member into hourly files, and recomputes every forecast of the four sizes from scratch on the laptop's CPU.