Lexical analysis  

From The Art and Popular Culture Encyclopedia

Jump to: navigation, search

Related e

Wiki Commons

Train wreck at Montparnasse (October 22, 1895) by Studio Lévy and Sons.
Train wreck at Montparnasse (October 22, 1895) by Studio Lévy and Sons.

In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. A program or function which performs lexical analysis is called a lexical analyzer, lexer, or scanner. A lexer often exists as a single function which is called by a parser or another function, or can be combined with the parser in scannerless parsing.

Lexical analyzer generators

  • ANTLR - Can generate lexical analyzers and parsers.
  • Flex - Alternative variant of the classic "lex" (C/C++).
  • JFlex - A rewrite of JLex.
  • Ragel - A state machine and lexical scanner generator with output support for C, C++, C#, Objective-C, D, Java, Go and Ruby source code.

The following lexical analysers can handle Unicode:

  • JavaCC - JavaCC generates lexical analyzers written in Java.
  • JLex - A lexical analyzer generator for Java.
  • Quex (or "Queχ") - A fast universal lexical analyzer generator for C and C++.

See also

Unless indicated otherwise, the text in this article is either based on Wikipedia article "Lexical analysis" or another language Wikipedia page thereof used under the terms of the GNU Free Documentation License; or on original research by Jahsonic and friends. See Art and Popular Culture's copyright notice.

Personal tools