Overview
Most notable lexer generators used in practice, including Flex, Ragel, and RE/flex are based on deterministic finite automata (DFA) for efficient pattern matching, despite the theoretical possibility of an exponential increase in DFA size. In practice, lexer specifications typically use deterministic regular expressions, which makes substantial DFA blowup uncommon.
RE/flex translates a POSIX-compliant lexer specification directly into a DFA using standard construction techniques described in the compiler literature, extending the techniques to handle lazy matching and indentation detection applicable to specific programming language tokenization tasks. Like Flex, RE/flex generates efficient DFA-based scanners, but it shares no code with Flex and is implemented as a complete rewrite in C++.
In addition to its native DFA-based engine, RE/flex can also be combined with external regular expression libraries that are not DFA-based, such as the C++ standard library regex engine, PCRE, and boost.regex. This is achieved by systematically rewriting the set of lexer patterns into a form suitable for tokenization with the selected external library. RE/flex performs this rewriting automatically using translation rules that are specific to each supported regular expression library. A lexer specification defines a set of regular expression patterns 
corresponding to different token classes, such as identifiers, keywords, literals, and operators. These patterns can be combined into a single regular expression 
. When applied to an input string, a regular expression engine repeatedly matches 
, returning the index i of the matched subpattern 
, thereby decomposing the input into a sequence of tokens.
Example use cases include:
