Overfitting Diagnostics

Every optimisation report shows the best result out of however many combinations were swept. What it never shows is the obvious follow-up question: how good a Sharpe ratio would the BEST of that many random, skill-free trials have produced anyway, just from trying enough of them? If the answer is "about as good as what you got," the optimisation found noise, not an edge - and nothing in a standard tester report tells you which one happened. This library implements the two statistics Marcos Lopez de Prado and David Bailey published specifically to answer that question. The Probabilistic Sharpe Ratio (PSR) converts an observed Sharpe ratio, its sample size, and the skewness and excess kurtosis of the underlying returns into the probability that the TRUE Sharpe ratio exceeds a chosen benchmark - it correctly widens the uncertainty band for fat-tailed, skewed return series instead of assuming returns are normal the way a naive Sharpe ratio calculation implicitly does. The Deflated Sharpe Ratio (DSR) then sets that benchmark to the Sharpe ratio you would EXPECT the best of N independent trials to reach by chance alone, using a closed-form approximation built on the Euler-Mascheroni constant. Feed it the number of parameter combinations actually tested and it tells you whether your best result clears the bar that many random attempts would have cleared anyway. A DSR comfortably above roughly 95% means the result still looks like skill after accounting for the search; a DSR near or below that means you cannot rule out that you simply tried enough things. Bonferroni and Holm-Bonferroni corrections are included for the case where you kept the full set of trial results rather than just the winner - a stricter, complementary check against declaring victory on the strength of one out of many hypothesis tests. One convention is easy to get backwards and is called out explicitly in every relevant function: this library works in EXCESS kurtosis (0.0 for a normal distribution), not raw kurtosis (3.0 for normal). Passing the wrong one shifts every downstream number without producing an obvious error. Notes and limitations - DiagExpectedMaxSharpe needs the standard deviation of Sharpe ratios ACROSS the trials you ran, which is only known exactly if you stored every trial's Sharpe. When you only kept the winner, the demo uses that single result's own standard error as a conservative stand-in - reasonable, but not the same as the real empirical spread, and it will understate deflation if your trials' Sharpes actually varied more than one trial's estimation noise suggests. Store the full optimisation report when you can. - All formulas assume independent trials. Parameter sweeps are rarely fully independent - neighbouring parameter values produce correlated results - which means the true "effective" number of trials is usually somewhat lower than the literal count, making DSR's warning conservative in the right direction (it treats correlated trials as if they were independent, which if anything makes the bar for "beats chance" too easy, not too hard, so an already-poor DSR should be trusted; a marginal pass is worth a second look). - The normal CDF and inverse CDF approximations used internally are accurate to roughly 1e-7 and 1e-9 respectively - more than sufficient for a diagnostic, not intended for pricing-grade precision work. - This library does not run a backtest, store trial results, or replace a proper walk-forward or out-of-sample test on either platform. It answers one narrow question: given the numbers you already computed, how much of the headline result could plausibly be chance.

Script · MT5 · v1.0 · Free

Tags: MT5 Script · MQL5 Source Code

MQL4 & MQL5 scripts · PolyMaria