TODO: check tail-call optimization in lexer implementation

[colin-kiegel]

https://doc.rust-lang.org/complement-design-faq.html#no-guaranteed-tail-call-optimization
https://mail.mozilla.org/pipermail/rust-dev/2013-April/003557.html

TODO:

• Check whether tail-calls are optimized?
• If not, what's the limit, before the lexer has to stop lexing?

[Nercury]

I think in worst case we can keep a depth counter of some sorts.