Skip to content

Commit 1b08e08

Browse files
ChristianStadelmannChristian Stadelmann AU0001TC
authored andcommitted
Tokenizer: Fix line continuation after punctuation
Prior to this change, any line ending with [punctuation + '...'], for example `||...`, would cause the tokenizer to fail.
1 parent 5b76d7f commit 1b08e08

File tree

1 file changed

+8
-0
lines changed

1 file changed

+8
-0
lines changed

tokenize_code.m

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,14 @@
9999
% any other operator:
100100
else
101101
symbol = skip(punctuation);
102+
% ends with '...':
103+
% The '...' has to be unskipped and handled here in order
104+
% to not cause and error for line endings such as `+...`
105+
% or `&&...`.
106+
if length(symbol) > 3 && strcmp(symbol(end-2:end), '...')
107+
pos = pos - 3;
108+
symbol = symbol(1:end-3);
109+
end
102110
% one operator:
103111
if any(strcmp(symbol, operators))
104112
add_token('punctuation', symbol);

0 commit comments

Comments
 (0)