Unicode Text Codec

StringToCharArray(text, array, 0, WHOLE_ARRAY, CP_UTF8) already converts to UTF-8 most of the time, and that built-in is the right tool for most jobs. What it does not give you is the codepoint boundary itself: there is no way to ask it where character five starts in the encoded bytes, or whether a byte buffer that came from a file, a socket or another program's output is even valid UTF-8 to begin with. Both questions need to walk the encoding by hand, one codepoint at a time - which is what mangled accents in files and requests usually trace back to: something upstream wrote text that was never actually valid UTF-8, and nothing downstream noticed until much later. The detail every hand-rolled UTF-8 decoder gets wrong at least once is overlong encoding. The same codepoint can often be written with more bytes than its minimal form needs - the null character encoded as two bytes instead of one, for instance. The specification requires the minimal form and forbids the rest for a real reason: overlong sequences have historically been used to smuggle a forbidden byte past a filter that only checks the short form of that character. This decoder rejects anything not in minimal form outright, along with truncated sequences, bad continuation bytes, and codepoints in the surrogate range, which are never valid UTF-8 on their own. The other detail is surrogate pairs. Both MQL4 and MQL5 strings are UTF-16 internally - StringGetCharacter reads one 16-bit code unit at a time on either platform - and any codepoint above the Basic Multilingual Plane, which covers most emoji, is stored as two code units that only mean something combined. A decoder that loops over StringGetCharacter one unit at a time without checking for a surrogate pair silently corrupts exactly that class of character. This header combines and splits surrogate pairs correctly on both the encode and decode paths, and Utf16CodepointCount() gives an honest character count where StringLen() would over-count by one for every such character. Notes and limitations - Utf8Decode() leaves result holding whatever was decoded before the failure point when it returns false - call IsValidUtf8() first if partial output on failure would be unsafe for the caller's purpose. - This validates and converts UTF-8 specifically. It does not detect or convert other encodings (Latin-1, Windows-1252, UTF-16LE files) - a file in one of those will simply fail IsValidUtf8() rather than being auto-detected and converted. - Utf8Encode() replaces an unpaired UTF-16 surrogate with U+FFFD rather than emitting the non-standard "WTF-8" encoding some tools use for it. A well-formed MQL string built through normal string operations should never actually contain an unpaired surrogate; this only matters if a caller constructs one deliberately or via a corrupted upstream source. - Codepoints above U+10FFFF are rejected by Utf8NextCodepoint(), matching the actual limit of the Unicode codepoint space, not the 4-byte format's older, wider theoretical limit from the original UTF-8 definition.

Script · MT5 · v1.0 · Free

Tags: MT5 Script · MQL5 Source Code

MQL4 & MQL5 scripts · PolyMaria