[Suprising behavior] lazy quantifiers are treated the same way as greedy quantifiers

[Gompyn]

I want to use interegular for tokenization in a C-like programming language. In particular, I use the regex \/\*.*?\*\/ to match the block comments. By using ?, I expect the lexer can immediately return a block comment when it see some string constitutes one.

from interegular import parse_pattern

fsm = parse_pattern(r'(?s:\/\*.*?\*\/)').to_fsm()
assert '/**/*/' not in fsm

However, this pattern is treated the same way as if we removed ?. This causes the string /**/*/ to wrongly match the regex, and by the 'most vexing' principle, this pattern eats every character even after a block comment finishes.

This can be seen from the source code that ? are simply ignored.

interegular/interegular/patterns.py

Lines 601 to 612 in 758f837

	if self.static_b("*"):
	if self.static_b("?"):
	pass
	return _Repeated(base, 0, None)
	elif self.static_b("+"):
	if self.static_b("?"):
	pass
	return _Repeated(base, 1, None)
	elif self.static_b("?"):
	if self.static_b("?"):
	pass
	return _Repeated(base, 0, 1)

[MegaIng]

I want to use interegular for tokenization in a C-like programming language.

interegular is completely the wrong library for that. Use a proper regex library.

lazy quantifiers are treated the same way as greedy quantifiers

Yes, by design. This will not be changed. These quantifiers don't actually affect which strings are capbable of being matched. interegular works somewhat as-if we are doing a fullmatch. And if you check, re.fulmatch(..., '/**/*/') does indeed match.

[Gompyn]

Is it possible to add a warning when detecting such patterns? I think this can greatly reduce the suprise and debugging difficulty for users that is using this library with lazy quantifiers. After all, the users mean something when they write a ? if they don't need to.

[MegaIng]

No. This library is not meant for such direct usages anyway, but as a tool for lark to detect overlapping regexes, so it tries to be compatible with stdlib re syntax.

[Gompyn]

While I guess this might affect regex collision detection in lark, I don't have any evidence for that. I'm OK if this issue is now simply closed.

[MegaIng]

Those warnings would be spurious in that context. Yes, people mean something when they write *? - but what they mean doesn't really matter for these purposes, so producing warnings for it is just clutter.

Again, interegular is not the correct library for whatever you are trying to do.

[Gompyn]

Again, interegular is not the correct library for whatever you are trying to do.

Thank you. I will find another library for my use cases.