How the models were tested
What was held out, what was measured and what failed when the Volatility Board, the card ranking and the Curated gates were tested on 2022 to 2026 data.
The question
Two outputs on the site are inputs to trading decisions. The Volatility Board gives a probability that each volatility metric moves over the next 21 sessions. The Options Dispatch shows short option cards and a Curated short list. Each output was tested against history before release. This page records what each test held out, what it measured and whether it passed.
The method
Every test used the same design.
- A kill condition was written before the run: the result that would reject the idea.
- Models were trained on the past and scored only on the following month or year, never on data used in training.
- Each result has a 90% interval from a block bootstrap over months. Entries overlap heavily, so the bootstrap resamples months, not trades.
- Entries whose life spanned a stock split were dropped. Before that filter, unadjusted splits had moved one pool mean from -0.055 to -0.167.
The trade tests replay short cards under one rule. Entry is the session after the card's date, at mid. Exit is the first session the structure's ask reaches 75% of entry credit, a 25% profit, else 14 calendar days after entry. Only cards with more than 30 days to expiry are entered. Profit and loss is in units of entry credit: 0.035 means 3.5% of the premium collected. Every card has equal weight.
Inputs and lag
The tests used a local copy of production data: 20 symbols, 2022-01-03 to 2026-07-28, about 1,149 daily chains per symbol. Every gate and feature used only data known at the card's date. The Volatility Board's target starts the session after that date. The replay books no commissions and fills at end-of-day prices.
How it was validated
Chance and the metric's own move
The test held out 30 monthly folds, 2024-01 to 2026-06. It measured the Spearman correlation between E-Move and the actual 21-session change. VRP scored 0.612, interval 0.560 to 0.659. IV scored 0.423, interval 0.335 to 0.499. Both passed. A later test confirmed the 21-session horizon: the VRP signal's correlation was 0.592 at 21 sessions and 0.485 at 63.
Every retrain runs five promotion gates. When the VRP model fails, the retrain is blocked. When the model for another metric fails, that column shows its level only. The calibration gate requires that at least 60% of calls at or above the 65% floor come true. The Brier score must be no worse than that of a constant base-rate forecast. The IV rise model failed this gate: 38 calls reached the floor, and 0.368 of them came true. Since 2026-09-04 the IV column shows its level only. A 2026-09-14 rework of the IV input also failed.
The Chance is calibrated on the metric's own move. No test found that it orders the profit of a naked trade.
Carry-win forecasts
A carry win means that 30-day implied volatility, earnings removed, exceeds historical volatility over the next 21 sessions. Four tests tried to forecast a carry win, each with its own kill condition. Each held out one calendar year at a time. All four had a Brier score worse than a constant base-rate forecast.
| Test | Brier | Base-rate Brier | Hit rate at or above 65% |
|---|---|---|---|
| 2026-08-31 V9 S0 | 0.2570 | 0.2533 | 0.546, all years |
| 2026-09-03 V9 S0R | 0.2573 | 0.2467 | 0.514, 2022 |
| 2026-09-13 V10 | 0.2664 | 0.2470 | 0.507, 2022 |
| 2026-09-13 V10B | 0.2825 | 0.2466 | 0.5325, 2022 |
The yearly share of carry wins was 0.49 in 2022, 0.68 in 2023 and 0.56 in 2024. No input known at the card's date tracked that shift, and the carry-win forecast was dropped.
Card fields and naked profit
The 2026-08-18 test replayed 76,232 entries. It measured the within-day Spearman correlation of each card field with naked profit. No field ranked naked profit.
| Field | Spearman | 90% interval |
|---|---|---|
| z(30d) | -0.010 | -0.026 to +0.004 |
| VRP points | -0.006 | -0.031 to +0.023 |
| IV percentile | +0.004 | -0.024 to +0.029 |
| Premium over spot | -0.103 | -0.124 to -0.081 |
Cards with a higher premium over spot had worse results. A win probability passed its calibration out of sample, but profit fell as that probability rose. Taking the top 5 or 10 cards a day by any field did worse than taking every card. In that replay a card was held to expiry when its target did not fill. The 14-day rule was introduced in the next test.
The 2026-09-15 test replayed the trade rule both delta hedged and naked. It tested the card ranking key, the order then used by the Curated view, built from the Volatility Board's IV E-Move. The key ordered hedged profit, Spearman 0.079, interval 0.023 to 0.133. It ordered naked profit the wrong way: -0.162, interval -0.213 to -0.104. Hedged, the pool lost 0.016 per card, interval -0.032 to -0.001. The traded book is naked, so the ranking key was retired. No score replaced it. The Curated view lists cards by symbol, then by expiry.
Pre-trade gates
The 2026-09-15 gate test used the same 76,225 naked entries, with a pool mean of -0.012. A gate passed when three conditions held:
- Its pass set beat the pool, with the interval clear of zero.
- It kept at least 10% of the pool.
- It beat the pool in both 2022 to 2023 and 2024 to 2026.
| Gate | Pass share | Pass minus pool | 90% interval | Verdict |
|---|---|---|---|---|
| Index and rates ETFs | 0.32 | 0.050 | 0.027 to 0.076 | pass |
| Symbol's own past record | 0.58 | 0.027 | 0.012 to 0.047 | pass |
| No earnings inside the hold | 0.90 | 0.007 | 0.002 to 0.013 | pass |
| Straddle, not strangle | 0.50 | 0.007 | -0.000 to 0.015 | fail |
| DTE 31 to 60 | 0.69 | -0.006 | -0.010 to -0.002 | fail, wrong way |
| No FOMC inside the hold | 0.65 | -0.003 | -0.015 to 0.009 | fail |
| VIX term in contango | 0.89 | -0.002 | -0.009 to 0.007 | fail |
| Market vol above its yearly mean | 0.44 | 0.019 | -0.009 to 0.041 | fail |
| IV percentile at or above 0.5 | 0.41 | -0.001 | -0.026 to 0.022 | fail |
| VRP above its trailing median | 0.61 | 0.006 | -0.013 to 0.028 | fail |
The three passing gates together kept 30% of the pool and averaged 0.035, interval 0.007 to 0.067. That set lost money only in 2024, at -0.019. The tested ETF list was SPY, QQQ, DIA, IWM and TLT. Every card on those five, before any other gate, averaged 0.038. The worst bucket in the study was a single-name earnings date 8 to 14 days inside the hold, at -0.120.
Curated uses the class gate and the earnings check. The symbol-record gate was dropped because it requires a running replay per symbol. Curated also keeps straddles only. Inside the class, straddles averaged 0.030 and strangles 0.046.
Long straddles before earnings
Two tests bought an at-the-money straddle 3 to 20 days before an earnings release and sold it before the release. The verdict used fills at bid and ask.
| Test | Expiry | Mean at bid and ask | 90% interval | At mid |
|---|---|---|---|---|
| 2026-09-15 V15 | 31 to 100 DTE | 0.019 | -0.001 to 0.038 | 0.040 |
| 2026-09-15 V16 | first after the print | 0.019 | -0.014 to 0.051 | 0.039 |
Both tests failed. The bid-ask spread cut the mean to about half of its value at mid, and both intervals reach below zero. The site has no long-vol card. Selling into the run-up was not tested.
Limits
- The data covers 20 symbols over four and a half years, one sequence of market regimes.
- No test books commissions. The short-card tests fill at end-of-day mid.
- Ten gates were tested at 90%. The both-halves condition does not correct for the number of gates tested.
- Curated's exact combination (class, earnings, straddle and DTE 28 to 60) was not replayed as a set with an interval.
- The curated class is a configured list. The replay tested only the five ETFs above.
- Two leads are untested: a market vol threshold above 1 standard deviation, and strangles on ETFs with straddles on single names.