Streaming Volatility: EWMA, RiskMetrics, and Online Estimators

1. The batch trap

Most volatility formulas are written as a pass over a fixed array of returns: sum the squares, divide, take a root. That is fine for a nightly report. In a live pipeline it fails twice. First on speed: if every incoming tick triggers a fresh pass over the last N returns, the work per tick grows with the window, and a system that must react in microseconds spends them re-adding numbers it already added a moment ago. Second on memory: keeping the whole history around just to recompute a single number is wasteful when the answer could be carried in a few running totals.

An online (or streaming) estimator flips the model. Instead of "here is all the data, compute the statistic," it holds a small, fixed-size state and exposes one operation — push(x) — that folds the newest observation into that state in constant time. Read the current estimate whenever you like. The whole point is that the cost per update does not depend on how much data has already streamed through.

2. EWMA: volatility with a fading memory

The most widely used streaming volatility estimator is the exponentially weighted moving average of squared returns, popularised by J.P. Morgan's RiskMetrics in the 1990s. It assumes returns are approximately zero-mean and updates variance with a single recursion:

σ²_t = λ · σ²_t−1 + (1 − λ) · r²_t

Every new squared return nudges the variance a little; every past squared return already baked in decays by a factor of λ each step. Unrolled, today's estimate is a weighted average of all past squared returns with geometrically declining weights — the recent past counts most, the distant past fades smoothly to nothing. Compared with a flat rolling window, EWMA has two advantages: it needs O(1) memory (one number, not a buffer of returns), and it has no hard edge — a big return doesn't abruptly drop out of the estimate N steps later, it just decays away.

3. Choosing the decay

The single parameter λ in (0, 1) sets how long the memory is. The cleanest way to reason about it is the half-life — how many observations until a shock's weight halves:

half‑life = ln(0.5) / ln(λ)

There is no universally correct value — it is the classic responsiveness-versus-stability trade. A smaller λ reacts to a regime change in a few observations but jitters on noise; a larger one is smooth and calm but slow to notice that the world has changed. RiskMetrics settled on 0.94 for daily data as a pragmatic middle; on an intraday tape sampled every few seconds you will generally want a shorter half-life. Whatever you pick, express it as a half-life first — it is far more intuitive than the raw decay, and it makes two different sampling frequencies comparable.

4. The numerical trap in "just sum the squares"

Suppose instead of exponential weighting you want an honest variance over the data seen so far. The textbook shortcut keeps two running totals — the sum and the sum of squares — and combines them at the end:

Var = ( Σ x² − (Σ x)² / n ) / (n − 1)

It is O(1) per update and looks perfect. It is also a well-known way to get a badly wrong answer. When the mean is large relative to the spread — a price hovering around 40,000 with tick-level wiggles, say — Σx² and (Σx)²/n are two enormous, nearly equal numbers, and subtracting them destroys most of the significant digits. This is catastrophic cancellation: the true variance is a tiny difference between two giants, and floating-point can even return a small negative number for a quantity that must be non-negative.

The fix is Welford's algorithm (1962), which updates the mean and the sum of squared deviations directly, so it never forms those giant intermediate sums:

δ = x − mean
mean ← mean + δ / n
M2 ← M2 + δ · (x − mean)

After n observations, the sample variance is simply M2 / (n − 1). Each update is a handful of operations, the state is three numbers, and because it works with deviations from the running mean it stays accurate no matter how large that mean is. It is the estimator to reach for whenever you need an exact running variance rather than an exponentially weighted one.

5. Rolling windows, done in constant time

Sometimes you want neither an all-history variance nor an exponential one, but the variance over exactly the last N observations — a hard trailing window. The naive version recomputes over the buffer on every tick; the better version updates incrementally. West's algorithm (1979) extends Welford with a remove step that is the exact inverse of the add, so a full window can evict its oldest value and absorb a new one in O(1), keeping a numerically stable mean and variance without ever rescanning the buffer.

The practical difference is stark on a busy tape. A 500-tick rolling variance recomputed each update does on the order of 500 operations per tick; the incremental version does a constant handful, whatever the window size. Over millions of ticks that is the gap between comfortably real-time and falling behind the feed.

6. Which estimator, when

The three tools answer three subtly different questions, and picking the right one matters more than tuning any of them.

Reach for EWMA as a live volatility gauge that tracks the current regime and forgets the past gracefully — it is the natural companion to a real-time risk limit. Use Welford when you want the exact, evenly-weighted variance of a whole session or since some reset. Use a rolling window when a metric is defined over a fixed lookback and every observation inside it should count equally. All three are model-free descriptions of recent variance; if you need a model that also mean-reverts volatility toward a long-run level and captures clustering explicitly, that is where a GARCH-family model earns its extra complexity — but for most live monitoring the streaming estimators here are faster, simpler, and quite enough.

7. Reading it honestly

A few cautions keep streaming volatility trustworthy. EWMA's zero-mean assumption is fine for high-frequency returns but drifts on data with a real trend — track the mean separately if it matters. Every estimate is only as clean as its returns: at very fine sampling, microstructure noise (bid-ask bounce, discreteness) inflates variance, so sampling by activity with information-driven bars often gives a steadier read than sampling by the clock. And a single number hides its own composition — recent variance driven by one jump should be read very differently from the same variance ground out by continuous diffusion, and the higher moments of realized skewness and kurtosis flag when the tails are doing the work. Streaming volatility is the fast, always-on backbone; read it beside those sharper lenses rather than instead of them.

8. Computing it

All three estimators are small, stateful objects — push one value at a time, read the current estimate. Our open-source orderflow-metrics library ships them, dependency-free, in TypeScript and Python, using Welford's and West's algorithms rather than the fragile sum-of-squares form:

import { Welford, EwmaVariance, RollingWindow } from "orderflow-metrics";

const vol = new EwmaVariance(0.94);   // RiskMetrics daily decay
for (const r of returns) vol.push(r);
vol.std;                              // current EWMA volatility

const w = new Welford();
for (const r of returns) w.push(r);
w.variance; w.mean;                   // exact running stats, O(1) per tick

const win = new RollingWindow(500);   // trailing 500-observation window
for (const r of returns) win.push(r);
win.variance;                         // O(1) add + evict (West 1979)

The Python distribution exposes the same classes in idiomatic form. Point them at a live return stream and each push costs the same handful of operations whether it is the tenth tick or the ten-millionth — a live volatility gauge that keeps up with the feed, one more lens in the market microstructure stack we build in the open. It reads naturally alongside the efficiency picture from the variance ratio and the regime signal from the Hurst exponent, and drops straight into a backtest or a live monitor.

9. Conclusion

Real-time risk lives or dies on estimators that update in constant time and stay honest under floating point. EWMA gives you a volatility with a fading memory and a single, interpretable dial — its half-life. Welford gives you an exact running variance that never loses precision to catastrophic cancellation. West's rolling window gives you an evenly-weighted lookback that evicts and absorbs in O(1). None of them rescans history, none of them hoards it, and together they turn "how volatile is the market right now?" into a question you can answer on every tick. Explore the rest of the toolkit in our quantitative research library, or read the implementation in our open-source metrics.