Class AbstractTokenizer

java.lang.Object
org.infosmid.stream.dsl.AbstractTokenizer
Direct Known Subclasses:
Tokenizer
public abstract class AbstractTokenizer extends Object
The base class for tokenizers. Knows about tokenization in general but not specific to a DSL variant. Provides state management and utility methods to real tokenizer subclasses.

  • Field Details

    • expressionString

      protected String expressionString
      The string to be tokenized.

    • toProcess

      protected char[] toProcess
      The expressionString as a char array.

    • max

      protected int max
      Length of input data.

    • pos

      protected int pos
      Current lexing position in the input data.

    • tokens

      protected List<Token> tokens
      Output stream of tokens.

    • linebreaks

      protected int[] linebreaks
      Positions of linebreaks in the parsed string.

  • Constructor Details

    • AbstractTokenizer

      public AbstractTokenizer()

  • Method Details

    • getTokens

      public Tokens getTokens(String inputData)

    • isTwoCharToken

      protected boolean isTwoCharToken(TokenKind kind)
      Check if this might be a two character token.
      Parameters:
      kind - the TokenKind to be evaluated
      Returns:
      true if the token has length of 2

    • pushCharToken

      protected void pushCharToken(TokenKind kind)
      Push a token of just one character in length.
      Parameters:
      kind - the TokenKind to be pushed

    • pushPairToken

      protected void pushPairToken(TokenKind kind)
      Push a token of two characters in length.
      Parameters:
      kind - the TokenKind to be pushed

    • isIdentifier

      protected boolean isIdentifier(char ch)

    • isQuote

      protected boolean isQuote(char ch)

    • isWhitespace

      protected boolean isWhitespace(char ch)

    • isDigit

      protected boolean isDigit(char ch)

    • isAlphabetic

      protected boolean isAlphabetic(char ch)

    • subArray

      protected char[] subArray(int start, int end)

    • lexIdentifier

      protected void lexIdentifier()

    • lexQuotedStringLiteral

      protected void lexQuotedStringLiteral()
      Lex a string literal which uses single quotes as delimiters. To include a single quote within the literal, use a pair ''.

    • lexDoubleQuotedStringLiteral

      protected void lexDoubleQuotedStringLiteral()
      Lex a string literal which uses double quotes as delimiters. To include a single quote within the literal, use a pair "".

    • isArgValueIdentifierTerminator

      protected boolean isArgValueIdentifierTerminator(char ch, boolean quoteOpen)
      For the variant tokenizer (used following an '=' to parse an argument value) we only terminate that identifier if encountering a small set of characters. If the argument has included a ' to put something in quotes, we remember that and don't allow ' ' (space) and '\t' (tab) to terminate the value.
      Parameters:
      ch - char to be evaluated
      quoteOpen - true if parsing an open quote
      Returns:
      true if char is a type of terminator and quote is not open.

    • lexArgValueIdentifier

      protected void lexArgValueIdentifier()
      To prevent the need to quote all argument values, this identifier lexing function is used just after an '=' when we are about to digest an arg value. It is much more relaxed about what it will include in the identifier.

    • sameQuotes

      protected boolean sameQuotes(int pos1, int pos2)

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • raiseException

      public void raiseException(DSLMessage message, Object... inserts)

    • addLinebreak

      protected void addLinebreak()