Class Tokenizer

java.lang.Object

org.infosmid.stream.dsl.AbstractTokenizer

org.infosmid.stream.dsl.Tokenizer

public class Tokenizer
extends AbstractTokenizer

Lex some input data into a stream of tokens that can then then be parsed.

Field Summary

Fields inherited from class AbstractTokenizer

expressionString, linebreaks, max, pos, tokens, toProcess

Constructor Summary

Constructors

Constructor	Description
Tokenizer()

Method Summary

Modifier and Type	Method	Description
protected void	process()
String	toString()

Methods inherited from class AbstractTokenizer

addLinebreak, getTokens, isAlphabetic, isArgValueIdentifierTerminator, isDigit, isIdentifier, isQuote, isTwoCharToken, isWhitespace, lexArgValueIdentifier, lexDoubleQuotedStringLiteral, lexIdentifier, lexQuotedStringLiteral, pushCharToken, pushPairToken, raiseException, sameQuotes, subArray

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

Tokenizer

public Tokenizer()

Method Details

process

protected void process()

toString

public String toString()

Overrides:

toString in class AbstractTokenizer