Historical Series Scrubber

A mean-and-standard-deviation outlier test is skewed by the very outliers it is trying to find - a single huge spike can inflate the mean and standard deviation enough that the spike itself no longer looks abnormal by the test's own numbers. This library uses a Hampel- filter-style test instead: for each point, it takes a local window of neighbouring values, computes their median and their median absolute deviation (MAD, scaled by 1.4826 to approximate a standard deviation under a normal distribution), and flags the point as an outlier only if it sits more than a configurable number of scaled-MAD units from that local median. Median and MAD are themselves robust to outliers, so a single spike in the window barely moves either statistic - unlike a mean/stdev test, the spike does not get to hide behind its own distortion of the yardstick used to judge it. Flagged points are replaced either with the local window median or, if configured, by interpolating between the nearest CLEAN (non-flagged) neighbours on either side - the interpolation path walks outward past any adjacent flagged points rather than interpolating against another value that was itself just judged an outlier, so a short run of consecutive bad points does not corrupt its own replacement. Every point the scrubber actually changes is recorded in an audit trail - index, timestamp, original value, replacement value, and the modified z-score (in MAD units) that triggered the change - so a caller can review, log, or reverse exactly what was touched rather than trusting a silent in-place rewrite. Notes and limitations - The window is clamped, not wrapped, at the series edges - the first and last few points are tested against a smaller, one-sided window, which makes them slightly less reliably judged than interior points. - A window where every value is identical (scaled MAD of exactly 0) treats ANY differing value as an outlier - correct in the strict statistical sense (zero local variation makes any deviation infinitely large in MAD units) but worth knowing about on a series with long flat stretches (e.g. an illiquid symbol with repeated identical prints). - If every point in the whole series is flagged as an outlier (a pathological, near-impossible but not-forbidden case), the interpolation replacement policy falls back to the local median instead, since there is no clean neighbour left to interpolate against. - The audit trail is capped at SCRUBBER_MAX_AUDIT_ENTRIES (2000 entries); HistoricalSeriesScrubberAppendAudit() returns false once full rather than growing without bound - fine for the scrubbing runs this library targets (bar-count history, not tick-by-tick series with a huge fraction of outliers). - This library performs no trading action and reads no live market data - it is pure array-transformation logic over a series the caller already holds.

Script · MT4 · v1.0 · Free

Tags: MT4 Script · MQL4 Source Code

MQL4 & MQL5 scripts · PolyMaria