Class Tokens

java.lang.Object
org.infosmid.stream.dsl.Tokens
public class Tokens extends Object
Class that converts an expression into a list of tokens. Furthermore, this class provides methods to process the tokens and keeps track of the current token being processed.

  • Method Details

    • getExpression

      public String getExpression()
      Return the expression string converted to tokens.
      Returns:
      expression string

    • decrementPosition

      public int decrementPosition()
      Decrement the current token position and return the new position.
      Returns:
      new token position

    • position

      public int position()
      Return the current token position.
      Returns:
      current token position

    • getTokenStream

      public List<Token> getTokenStream()
      Return an immutable list of tokens
      Returns:
      list of tokens

    • lookAhead

      protected boolean lookAhead(int distance, TokenKind desiredTokenKind)
      Return true if the token in the position indicated by position + distance matches the token indicated by desiredTokenKind.
      Parameters:
      distance - number of token positions past the current position
      desiredTokenKind - the token to check for
      Returns:
      true if the token at the indicated position matches desiredTokenKind

    • hasNext

      public boolean hasNext()
      Return true if there are more tokens to process.
      Returns:
      true if there are more tokens to process

    • peek

      public Token peek()
      Return the token at the current position. If there are no more tokens, return null.
      Returns:
      token at current position or null if there are no more tokens

    • peek

      protected Token peek(int howFarAhead)
      Return the token at a specified distance beyond current position. If that is off the end of the list of known tokens, return null.
      Parameters:
      howFarAhead - how many character positions to peek ahead.
      Returns:
      token at specified distance beyond current position or null if there are no more tokens

    • peek

      public boolean peek(TokenKind desiredTokenKind)
      Return true if the indicated token matches the current token position.
      Parameters:
      desiredTokenKind - token to match
      Returns:
      true if the current token kind matches the provided token kind

    • next

      public Token next()
      Return the next Token and advance the current token position.
      Returns:
      next Token

    • eat

      public Token eat(TokenKind expectedKind)
      Consume the next token if it matches the indicated token kind; otherwise throw CheckPointedParseException.
      Parameters:
      expectedKind - the expected token kind
      Returns:
      the next token
      Throws:
      CheckPointedParseException - if the next token does not match the expected token kind

    • eat

      protected Token eat()
      Consume the next token. Throw CheckPointedParseException if there is nothing to consume.
      Returns:
      the next token
      Throws:
      CheckPointedParseException - if there are no more tokens

    • isNextAdjacent

      public boolean isNextAdjacent()
      Return true if the first character of the token at the current position is the same as the last character of the token at the previous position.
      Returns:
      true if the first character of the current token matches the last character of the previous token

    • checkpoint

      protected void checkpoint()
      Indicate that the current token has been successfully processed.
      See Also:
      • lastGoodPosition

    • raiseException

      public void raiseException(int position, DSLMessage message, Object... inserts)
      Throw a new CheckPointedParseException based on the current and last successfully processed token position.
      Parameters:
      position - position where parse error occurred
      message - parse exception message
      inserts - variables that may be inserted in the error message

    • getLine

      public int getLine(Token token)
      Parameters:
      token - the token for which to determine the line
      Returns:
      which line the token is on, starting from 0