Tokenizer And Lexer Kit

Any program that parses a small custom syntax - a risk-rule expression, a config line, a formula typed into an input field - starts with the same job: split the raw text into tokens before any grammar can be applied to them. Getting that scanning stage right by hand, every time, is where subtle bugs creep in, and two of them recur constantly. The first is greedy multi-character operator matching. A naive character-by-character scan sees "==" as two "=" tokens and "<=" as "<" then "=" unless it specifically checks for the two-character form first. This library's operator table is ordered longest-match-first and always returns the longest operator that matches at the current position, so "==", "!=", "<=", ">=", "&&", "||" and the compound assignment operators are never split. The second is distinguishing a unary sign from a binary operator. "a-1" should tokenize as identifier, minus-operator, number - but "x*-3" should tokenize the "-3" as one signed number token, because a minus right after another operator, an open bracket, or the very start of input can only be a sign, never a subtraction. The lexer tracks the previous token's type to make this call the same way a hand-written recursive-descent parser's grammar would, rather than guessing from the character alone. Numbers accept an optional decimal point and an optional exponent (1.5e-3), strings accept both single and double quotes with backslash escapes (\n, \t, \r, \\, \" , \'), and identifiers follow the usual letter-or-underscore-start rule. Anything that matches nothing recognised becomes a single-character TOK_UNKNOWN token rather than aborting the scan - a caller building a parser on top decides what to do with unexpected input, this layer's job is only to classify what it can and hand back everything it saw. Notes and limitations - Operators are matched from a fixed table (see g_TokOperators in the header); adding a new multi-character operator means adding it to that table BEFORE any of its own single-character prefixes, since the table is scanned in order and the first match wins. - The unary-sign rule only fires directly in front of a number literal ("-3", "+2.5e2"); a sign in front of an identifier or an opening parenthesis ("-x", "-(a+b)") is correctly tokenized as a plain TOK_OPERATOR minus, for the calling parser's grammar to handle as unary itself. - This is a lexer, not a parser - it has no concept of operator precedence, matching brackets, or grammar rules. It only classifies and positions tokens; building an expression tree from them is a separate layer. - String escape decoding covers \n \t \r \\ \" \' ; any other backslash sequence passes the following character through literally rather than raising an error.

Script · MT4 · v1.0 · Free

Tags: MT4 Script · MQL4 Source Code

MQL4 & MQL5 scripts · PolyMaria