Skip to content

Commit e71780e

Browse files
ChristianStadelmannChristian Stadelmann AU0001TC
authored andcommitted
Tokenizer: Fix line continuation after punctuation
Prior to this change, any line ending with [punctuation + '...'], for example `||...`, would cause the tokenizer to fail.
1 parent 5b76d7f commit e71780e

1 file changed

Lines changed: 6 additions & 0 deletions

File tree

tokenize_code.m

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,12 @@
9999
% any other operator:
100100
else
101101
symbol = skip(punctuation);
102+
% ends with '...'
103+
if length(symbol) > 3 && endsWith(symbol, '...')
104+
% unskip '...'
105+
pos = pos - 3;
106+
symbol = symbol(1:end-3);
107+
end
102108
% one operator:
103109
if any(strcmp(symbol, operators))
104110
add_token('punctuation', symbol);

0 commit comments

Comments
 (0)