Multiple Testing Overfit Demo
"I tried a hundred parameter combinations and this one made the most money" sounds like research. It is usually the multiple-testing lottery: test enough random configurations against the same stretch of history and one of them will look good by chance alone, with no genuine edge behind it. This script demonstrates that directly rather than asserting it. A batch of random fast/slow moving-average period pairs is tested on an older, in-sample window of this chart's history using a simple always-in- the-market crossover rule (long when the fast MA sits above the slow MA, short otherwise), with every signal read from the PRIOR bar's MAs so the backtest never looks ahead. Whichever pair scored best in-sample is crowned, then tested again - same parameters, untouched - on the more recent out-of-sample window that follows. The number that actually proves the point is not just "in-sample beat out-of-sample". A real edge is also expected to decay somewhat out of sample. What exposes pure overfitting is the crowned trial's RANK among the whole field once everyone is re-sorted by out-of-sample return instead of in-sample return: if the "best" in-sample performer turns out to be merely average, or worse, out-of-sample, its in-sample win was noise that happened to line up with that window, not skill that transfers. Notes and limitations - This is a teaching demonstration, not a strategy or a strategy-selection tool. The crossover rule used is deliberately simple (always in the market, no costs, no stops) so the selection-bias effect is the only thing being measured. - No spread, commission or slippage is modelled; the point/price return figures are gross, comparative numbers between trials, not a realistic P/L estimate. - Results depend on the specific history window loaded on the chart and on the random seed for that run - relaunching with the same chart can and will produce a different crowned trial. - Requires enough bars loaded on the chart (in-sample + out-of-sample + the longest possible slow MA period, plus one); it checks this and reports plainly if there is not enough.
Script · MT5 · v1.0 · Free
About Multiple Testing Overfit Demo
Multiple Testing Overfit Demo is a script for MetaTrader 5. It can alert you with a pop-up, sound, email or push message. It has 7 settings, such as Trials, Fast period min and Fast period max. MQL source code included.
Files in the download: MultipleTestingOverfitDemo.ex5, MultipleTestingOverfitDemo.mq5, MultipleTestingOverfitDemo.txt.
Settings
| Setting | Default | What it does |
|---|---|---|
| Trials | 120 | Random fast/slow MA parameter pairs to test |
| Fast period min | 3 | Fast MA period, random range lower bound |
| Fast period max | 20 | Fast MA period, random range upper bound |
| Slow period min | 21 | Slow MA period, random range lower bound |
| Slow period max | 100 | Slow MA period, random range upper bound |
| In sample bars | 300 | Bars in the in-sample (fitting) window, older history |
| Out sample bars | 150 | Bars in the out-of-sample (test) window, most recent closed bars |
How to install
- Download the ZIP from your PolyMaria library and unzip it.
- In MetaTrader 5 open File → Open Data Folder, then go to MQL5 → Scripts.
- Copy the .mq5 file(s) there. Put any .mqh files in MQL5 → Include.
- Open the .mq5 file in MetaEditor and press Compile (F7) if MetaTrader does not show it.
- Drag the script onto a chart to run it once.
Tags: Moving Average · Alerts · MT5 Script · MQL5 Source Code