Columnar Series File Format
A row-oriented export (one struct-shaped record per row, the usual approach) has an awkward cost profile for selective reads: pulling just the Close column out of a million-row OHLCV history still means reading past every Open/High/Low/Volume/Time field along the way, because they are physically interleaved with the one field actually wanted. That cost scales with the FULL row width, not with the one column being read. This library stores data the other way round. Every declared column is written as one contiguous block of same-type values, and a small header directory up front records exactly where each column's block starts. Reading a whole column is one seek straight to its offset followed by one bulk read of rowCount values - every other column's bytes are never touched. Reading a SINGLE cell is even cheaper: offset plus rowIndex times 8 bytes, one seek, one read - no need to load the column at all, let alone the file. The trade-off is the opposite of a row store: appending a new row would require touching every column's block (shifting all of them along), so this format is meant for data written once in bulk - an exported price/indicator history - then read selectively many times, not for a live, growing dataset. Notes and limitations - Only two fixed-width column types are supported: CSF_TYPE_DOUBLE and CSF_TYPE_LONG (both 8 bytes/value). There is no variable-length string column type - a string field would need a separate, parallel fixed-width or offset-indexed scheme, which this library does not provide. - The row count is fixed at Begin() and cannot grow afterwards without rewriting the whole file - this is a bulk-write, many-read format, not an append log (see AtomicFileSwapWriter/AppendOnlyJournal elsewhere in this catalogue for append-oriented needs). - WriteColumnDoubles()/WriteColumnLongs() must be called for every declared column with exactly rowCount values; a declared column that is never written leaves an unwritten (effectively garbage/zero) gap in the file rather than raising an error at Close() time.
Script · MT5 · v1.0 · Free
About Columnar Series File Format
Columnar Series File Format is a script for MetaTrader 5. MQL source code included.
Files in the download: ColumnarSeriesFormat.mq5.mqh, ColumnarSeriesFormat.txt, ColumnarSeriesFormatDemo.ex5, ColumnarSeriesFormatDemo.mq5.
How to install
- Download the ZIP from your PolyMaria library and unzip it.
- In MetaTrader 5 open File → Open Data Folder, then go to MQL5 → Scripts.
- Copy the .mq5 file(s) there. Put any .mqh files in MQL5 → Include.
- Open the .mq5 file in MetaEditor and press Compile (F7) if MetaTrader does not show it.
- Drag the script onto a chart to run it once.
Tags: Volume · Binary Options · MT5 Script · MQL5 Source Code