Duplicate Row Remover
A file built by appending to it across several script runs or several terminal sessions accumulates duplicate rows the moment the same period gets logged twice - a script restarted after a crash, a backtest export re-run over an overlapping date range, a ticket that gets written again after being modified. The result opens fine in a spreadsheet, but any aggregation over it double-counts the repeated rows. The naive fix always keeps the first occurrence and throws away the rest. That is wrong exactly when the later run is the one with the corrected data - a ticket row re-logged after its close price was updated, for instance, where the first copy is now stale. This script lets you choose which copy survives: first-seen, for a log where duplicates are pure repeats, or last-seen, for one where a later write can supersede an earlier one for the same key. Rows are matched on one column only, read as raw text up to the next delimiter rather than parsed and reformatted, so a numeric key like a ticket number and a text key like a timestamp both work without any special-casing, and the surviving rows are written out byte-for-byte as they appeared in the source file. Notes and limitations - Duplicate detection compares the key column's raw text exactly. "100" and "100.0" are treated as different keys, since the column is never parsed as a number. - Rows are compared with an O(n squared) scan - simple and fast enough for the row counts a terminal export actually produces, but a file running into the hundreds of thousands of rows will take noticeably longer than a linear pass would. - Every line is copied through exactly as read, including any quoting a spreadsheet added on export - this script never re-serialises a row, it only decides whether to keep it. - Blank trailing lines at the end of the source file are dropped silently rather than counted as empty duplicate rows. - Does not reorder or filter columns - see CSV Column Reorder And Filter for that, and run this script either before or after it depending on which matters more for your pipeline.
Script · MT4 · v1.0 · Free
About Duplicate Row Remover
Duplicate Row Remover is a script for MetaTrader 4. It can alert you with a pop-up, sound, email or push message. It has 6 settings, such as Input file, Output file and Key column index. MQL source code included.
Files in the download: DuplicateRowRemover.ex4, DuplicateRowRemover.mq4, DuplicateRowRemover.txt.
Settings
| Setting | Default | What it does |
|---|---|---|
| Input file | Source CSV file, in MQL4\Files | |
| Output file | Destination CSV filename, in MQL4\Files | |
| Key column index | 1 | 1-based column used to detect duplicates |
| Has header | true | First row is a header, always kept |
| Keep first | true | Keep the first occurrence of each key (false = keep the last) |
| Delimiter | DelimiterComma | CSV column separator (source and destination) |
How to install
- Download the ZIP from your PolyMaria library and unzip it.
- In MetaTrader 4 open File → Open Data Folder, then go to MQL4 → Scripts.
- Copy the .mq4 file(s) there. Put any .mqh files in MQL4 → Include.
- Open the .mq4 file in MetaEditor and press Compile (F7) if MetaTrader does not show it.
- Drag the script onto a chart to run it once.
Tags: Trading Sessions · Alerts · MT4 Script · MQL4 Source Code