Comment support for parser

[FriedrichFroebel]

Currently, there seem to be two different lexers: the normal one and a full one (which preserves comments for example). There is only a normal parser, so I would have to use the full lexer if I want to work with the comments later. Having a full parser which supports comments would be easier to use. (Using parser.parse(infile.read(), lexer=full_lexer.clone()) throws an SyntaxError on <?php which does not occur when calling a full_lexer.clone() instance directly without the parser.)

[viraptor]

It's not that there are two different lexers: FilteredLexer provides some fixups that are harder in the parser and is required.

The parser itself doesn't support comments at all at the moment. Unfortunately I don't have plans to implement it. I'll be happy to accept PRs that do it.

[FriedrichFroebel]

Based on the phply modifications in tk.phpautodoc, the following code should enable a basic parser with comment support (the line numbers may be wrong, but this should not be a problem at the moment).

Currently this modifications are done directly in the existing code, but it probably would be better to use two parsers (one with comment support and one without, as there is an issue comment that a full-blown parser might we overkill in simple situations). I could not yet figure out how to integrate the code to let both parser options work side-by-side - otherwise I probably would already have opened a PR for it.

---
 phply/phplex.py   |  6 +++---
 phply/phpparse.py | 11 +++++++++++
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/phply/phplex.py b/phply/phplex.py
index 93b3f35..2251299 100644
--- a/phply/phplex.py
+++ b/phply/phplex.py
@@ -82,9 +82,6 @@ unparsed = (
 
     # Open and close tags
     'OPEN_TAG', 'OPEN_TAG_WITH_ECHO', 'CLOSE_TAG',
-
-    # Comments
-    'COMMENT', 'DOC_COMMENT',
 )
 
 tokens = reserved + unparsed + (
@@ -133,6 +130,9 @@ tokens = reserved + unparsed + (
 
     # Backtick
     'BACKTICK',
+
+    # Comments
+    'COMMENT', 'DOC_COMMENT',
 )
 
 # Newlines
diff --git a/phply/phpparse.py b/phply/phpparse.py
index 910fa75..7f22c9a 100644
--- a/phply/phpparse.py
+++ b/phply/phpparse.py
@@ -55,6 +55,7 @@ else:
 
 # Get the token map
 tokens = phplex.tokens
+ast.Comment = ast.node('Comment', ['text'])
 
 precedence = (
     ('left', 'INCLUDE', 'INCLUDE_ONCE', 'EVAL', 'REQUIRE', 'REQUIRE_ONCE'),
@@ -143,6 +144,7 @@ def p_top_statement_list(p):
 
 def p_top_statement(p):
     '''top_statement : statement
+                     | comment
                      | function_declaration_statement
                      | class_declaration_statement
                      | HALT_COMPILER LPAREN RPAREN SEMI'''
@@ -215,6 +217,7 @@ def p_inner_statement_list(p):
 
 def p_inner_statement(p):
     '''inner_statement : statement
+                       | comment
                        | function_declaration_statement
                        | class_declaration_statement
                        | HALT_COMPILER LPAREN RPAREN SEMI'''
@@ -659,6 +662,7 @@ def p_class_statement_list(p):
 
 def p_class_statement(p):
     '''class_statement : method_modifiers FUNCTION is_reference STRING LPAREN parameter_list RPAREN method_body
+                       | comment
                        | variable_modifiers class_variable_declaration SEMI
                        | class_constant_declaration SEMI
                        | USE fully_qualified_class_name LBRACE trait_modifiers_list RBRACE
@@ -672,6 +676,8 @@ def p_class_statement(p):
             p[0] = ast.TraitUse(p[2], [], lineno=p.lineno(1))
         else:
             p[0] = ast.ClassVariables(p[1], p[2], lineno=p.lineno(3))
+    elif len(p) == 2:
+        p[0] = p[1]
     else:
         p[0] = ast.ClassConstants(p[1], lineno=p.lineno(2))
 
@@ -1660,6 +1666,11 @@ def p_encaps_var_offset_variable(p):
     'encaps_var_offset : VARIABLE'
     p[0] = ast.Variable(p[1], lineno=p.lineno(1))
 
+def p_comment(p):
+    '''comment : COMMENT
+               | DOC_COMMENT'''
+    p[0] = ast.Comment(p[1])
+
 def p_empty(p):
     'empty : '
     pass
--