首页 > AI前沿 > Capability Emergence Can Be Forecast: Per-Seed, In Advance, With Calibrated Intervals, Certified False Alarms, and a Blind Pre-Registered Gate

Capability Emergence Can Be Forecast: Per-Seed, In Advance, With Calibrated Intervals, Certified False Alarms, and a Blind Pre-Registered Gate

arXiv机器学习 2026-07-18 04:54 3 阅读 查看原文

Emergent capabilities are widely treated as unpredictable: loss improves smoothly while abilities appear abruptly.

Prior work offers early-warning indicators but never scores them as forecasts: no lead time at controlled false-alarm rate, no calibration, no negatives, no blind tests.

We supply that discipline and show that, in grokking model systems and small language models, emergence timing is forecastable per run, in advance, with calibrated uncertainty.

Experiments and Results

Across 30 transformers at identical configuration, the formation time of the previous-token head forecasts each seed's induction-head emergence at Spearman rho=0.977 with median lead 975 steps (~15% of training); a best-case loss rule ties the ranking with 50-step lead (a nowcast).

Conformal intervals covered 15/15 held-out seeds, and the frozen rule passed blind pre-registered gates on TWO never-seen configurations (10/10 and 9/10 coverage).

A trap-language rung then attacked our own rule as pre-registered: where previous-token context pays for the task itself, the bare precursor false-alarms on 10/10 capability-blocked runs, while the mechanism-composed conjunction is certified in both language classes (0 false alarms) and times emergence at rho=1.000.

Finally, a gap-origin study broke the fixed offset (both lr and batch move the gap ~2.3x; no external clock owns it) and revealed the law beneath: across 80 valid-anchor runs the anchor fires at 0.843 of time-to-emergence -- t_event ~= 1.19 x t_anchor -- and this multiplicative rule passed its own blind gate (5/5) at a third unseen configuration.

False alarms are certified against 33 manufactured negatives.

The precursor leads across 3 public model families (Pythia, OLMo, OLMo-2; 7 suites), with OLMo-2 at 1B tokens showing precursor formed, capability absent.

Four pre-registered kill criteria fired and are reported.

Every freeze precedes its data in a public commit chain.