[Feature request] Smarter macros

[SkyyySi]

The current string-based macros, while powerfull, are also very difficult to actually use (at least in any way that couldn't just be done by a function). My suggestion would thus be to implement a simplified macro syntax which behaves more like Rust's declarative macros.

The idea is that one could write code like this:

macro typed_func = ( ( $( $arg:ident : $type:expr $( = $default_value:expr )? ),* $(,)? ) $arrow:token $body:block ) ->
    ( $(arg $( = $default_value )? ),* ) $arrow
        $(
        assert(type($arg) == $type)
        )*
        $block

--- Usage example:
f = $typed_func((x: "number", y: "number") -> x + y)
--- Expansion:
f = (x, y) ->
    assert(type(x) == "number")
    assert(type(y) == "number")
    x + y

Now, it would be very difficult (if not impossible) to actually implement this with the current parser. I would suggest the following solution to that: Whenever the parser hits a macro call, it could be made to parse a very loose subset of the language. It would essentially just need to be able to handle strings and comments - everything else does not get parsed, but read as literal text. Then, the macro gets used to generate new parser rules. Finally, the macro call is parsed with the now-extended parser.

This would by far be the easiest solution, but it would have the downside of requireing parenthesis around the macro's arguments. I would argue that two extra characters are a good tradeoff to make for getting a significantly more useful feature in return.

Alternatively, if the parsing library used by the YueScript compiler supports injecting new rules on-the-fly, parsing could also be done by making each macro definition immediatley modify the parser.

[pigpigyyy]

How about introducing two small syntax upgradings first?

Currently we can write macros like this:

macro printNumAndStrPair = (num, str) ->
  unless $is_ast(Num, num) and $is_ast String, str
    error "got unexpected token"
  "
print(
  #{num}
  #{str}
)
"

We put the AST token check right in the arguments and add a new string syntax from YAML. And then rewrite the code this way:

macro printNumAndStrPair = (num `Num, str `String) -> |
  print(
    #{num}
    #{str}
  )

[SkyyySi]

Those are some great improvements! I'm especially glad that it's no longer needed to manually quote and escape everything with the new unquoted string syntax. That part drove me insane.

I also like how this allows combining declarative and procedural macros (compared to how they are entirely different in Rust, with procedural macros being much more of a hassle to make), even allowing them to be mixed in one and the same macro. Thank you a lot for adding this!

About the "type" / syntax annotations: I'm guessing that they are currently implemented as purely validation, without changing the parsing in any way. Is that correct?

I don't know if that's possible with the parsing library used by YueScript, but maybe it could be made to have a macro definition inject a new parser node as soon as it gets parsed? Like having the parser immediately run a method as soon as it sees something like

macro foo = (x `String) ->
    -- ...

so that any occurrence of $foo after that is not handled by the regular function parsing node, but instead by that new node.

Anyway, I wouldn't consider that to be crucial. Since almost everything in YueScript is valid as an expression anyway, there are already very few limits to what can be passed in. And with the new unquoted/YAML-like strings, it's also fairly easy to have fully arbitrary input ($html macro anyone?) without the visual "noise" of using [[ ... ]].

The last thing I'm personally still missing would be the ability to have repeating groups. Parsing any sort of repeating (and potentially delimited) syntax is still a big hassle. Even something seemingly simple, like a comma-separated list of variable identifiers, requires significant effort to do, especially when not using an external pattern-matching library (since Lua's string patterns really struggle with grouping and repetition).

[pigpigyyy]

The "type" / syntax annotations are implemented as a syntax rule for all function definition but getting handled only in macro functions. You can find

f = (a`AnyTypeName) -> print a

is a valid syntax for the moment, but has no effect.

As for “injecting a new parser node”, maybe we can currently make use of the Lua code inserter macro.

macro foo = (input) ->
  return
    code: ... -- parse input text with self defined compiler and generate to Lua code here
    type: "lua"

print $foo -- If the generated Lua code is a valid expression, this will pass
$foo! -- if the generated Lua code has valid statements, it works too

Manipulating the YueScript inner C++ AST node might not be an easy work. I still suggest operating on the string of code and doing the syntax check manually with the builtin $is_ast() macro.

As for the repeating groups request. Not getting a good idea about a new syntax yet, but I just discovered somehow better readable ways to do it for now.

cat = (tb, sep = ', ') -> table.concat tb, sep

items = [1, 2, 3]

print |
  do
    print #{cat items}
    #{cat (for item in *items do |
      print "#{item}"
    ), '\n  '}

Output:

do
  print 1, 2, 3
  print "1"
  print "2"
  print "3"

[SkyyySi]

While playing around with the new syntax some more, I noticed one fairly annoying thing: when interpolating a multi-line statement that requires indentation is interpolated into a string, it will throw a syntax error if the interpolated code isn't indented to match the code it is being interpolated into.

Example:

macro foo = () ->
    const example_statement = |
        do
            print("Hello")
            print("World")

    |
        do
            -- Will throw a compiler error "failed to expand macro as expr: unexpected indent"
            #{example_statement}

            -- Correct version:
            #{example_statement::gsub("\n", "\n\t")}
            -- The number of tabs added above must match the indentation of the enclosing code.

print($foo())

Would it be possible to have the indentation be automatically adjusted instead?

---

Additionally, I noticed that commenting out code (at least in in YAML-like strings) seems to have some weird behaviour. If you comment out #{example_statement} in the above code example, the generated code will look like this:

return print((function()
    print("Hello")
    print("World")
    do
        print("Hello")
        return print("World")
    end
end)())

What's happening is that --#{example_statement} gets expanded immediately, thus making the code look like this to the compiler:

do
    --do
    print("Hello")
    print("World")

    do
        print("Hello")
        print("World")

Maybe comments should just be ignored in YAML-like strings?

[pigpigyyy]

Would it be possible to have the indentation be automatically adjusted instead?

This will require doing \gsub("\n", "\n\t") on every string interpolation for dealing possible \n. Maybe we should not alter the semantic of #{abc} but use a new string interpolation for it.

What's happening is that --#{example_statement} gets expanded immediately, thus making the code look like this to the compiler:

The indented text following the | symbol are all strings. Reserving the comment after | will break the rules and may make user think of its the real code.