scanToken

Scan for the next token and return it

Information

This information is part of the Modelica Standard Library maintained by the Modelica Association.

Syntax

(token, nextIndex) = Strings.scanToken(string, startIndex, unsigned=false);

Description

Function scanToken scans the string starting at index "startIndex" and returns the next token, as well as the index directly after the token. The returned token is a record that holds the type of the token and the value of the token:

token.tokenType Type of the token, see below
token.real Real value if tokenType == TokenType.RealToken
token.integer Integer value if tokenType == TokenType.IntegerToken
token.boolean Boolean value if tokenType == TokenType.BooleanToken
token.string String value if tokenType == TokenType.StringToken/IdentifierToken/DelimiterToken

Variable token.tokenType is an enumeration (emulated as a package with constants) that can have the following values:

import T = Modelica.Utilities.Types.TokenType;
T.RealToken Modelica Real literal (e.g., 1.23e-4)
T.IntegerToken Modelica Integer literal (e.g., 123)
T.BooleanToken Modelica Boolean literal (e.g., false)
T.StringToken Modelica String literal (e.g., "string 123")
T.IdentifierToken Modelica identifier (e.g., "force_a")
T.DelimiterToken any character without white space that does not appear
as first character in the tokens above (e.g., "&")
T.NoToken White space, line comments and no other token
until the end of the string

Modelica line comments ("// ... end-of-line/end-of-string") as well as white space is ignored. If "unsigned=true", a Real or Integer literal is not allowed to start with a "+" or "-" sign.

Example

import Modelica.Utilities.Strings;
import T = Modelica.Utilities.Types.TokenType;
(token, index) := Strings.scanToken(string);
if token.tokenType == T.RealToken then
   realValue := token.real;
elseif token.tokenType == T.IntegerToken then
   integerValue := token.integer;
elseif token.tokenType == T.BooleanToken then
   booleanValue := token.boolean;
elseif token.tokenType == T.Identifier then
   name := token.string;
else
   Strings.syntaxError(string,index,"Expected Real, Integer, Boolean or identifier token");
end if;

Syntax

(token, nextIndex) = scanToken(string, startIndex, unsigned)

Inputs (3)

string

Type: String

Description: String to be scanned

startIndex

Default Value: 1

Type: Integer

Description: Start scanning of string at character startIndex

unsigned

Default Value: false

Type: Boolean

Description: = true, if Real and Integer tokens shall not start with a sign

Outputs (2)

token

Type: TokenValue

Description: Scanned token

nextIndex

Type: Integer

Description: Index of character after the found token; = 0, if NoToken