Lexical Analyzer should recognize whitespace as a separate token

[MaikelH]

Problem

Whitespace characters are considered as separate tokens. This means the index positions do not always correspondents to the token locations.

Example

Take the expression 4 ?2*2

The lexical will throw an exception at location 1 that an unrecognized character is found and will display the space when this should be the question mark.

Solution

The lexical analyzer should ignore whitespace.