[enh] Auto-generate wrapper

[Technologicat]

On some occasions, a macro needs to invoke another:

@macros.expr
def aif(tree, gen_sym, **kw):
    # no hq[], the whole point of "aif" is to leak "it".
    bindings = [q[(it, ast_literal[test])]]
    body = q[ast_literal[then] if it else ast_literal[otherwise]]
    return _let(body, bindings, "let", gen_sym)  # <-- needs let[]

Or itself:

@macros.expr
def cond(tree, **kw):
    return _cond(tree)

def _cond(tree):
    elts = tree.elts
    if len(elts) == 1:  # final "otherwise" branch
        return elts[0]
    if not elts:
        assert False, "Expected cond[test1, then1, test2, then2, ..., otherwise]"
    test, then, *more = elts
    return hq[ast_literal[then] if ast_literal[test] else ast_literal[_cond(more)]]

This leads to boilerplate. To be able to, from inside a macro, "call" another macro, which may be defined in another module, it is (AFAIK, IIUC) currently necessary to split the definition into two parts:

• Decorated wrapper, for use by normal run-time code
• Syntax transformer function, for use from inside a macro

Having to split at all is slightly inelegant, but bearable; having to do it manually is too much typing. :)

Hence, enhancement suggestion: is it possible to automate this? I would imagine that the macro decorator could do the definition shuffling, and save the syntax transformer function (i.e. the original function being decorated) into an attribute of the macro name, e.g. .transform (unless this somehow doesn't fit the MacroPy implementation; my mental model here is regular decorators for regular functions).

This would shorten the second example to:

@macros.expr
def cond(tree, **kw):
    elts = tree.elts
    if len(elts) == 1:  # final "otherwise" branch
        return elts[0]
    if not elts:
        assert False, "Expected cond[test1, then1, test2, then2, ..., otherwise]"
    test, then, *more = elts
    return hq[ast_literal[then] if ast_literal[test] else ast_literal[cond.transform(more)]]

eliminating the explicit wrapper.

More use cases to follow later; I'll have to dig PG's On Lisp a bit for concrete examples where macros need to use other macros. (Racket, on the other hand, is in a large part built via macros on top of macros on top of...)

[azazel75]

See b7fd7c4

[Technologicat]

Wow. unpythonic.syntax and its 25 now obsoleted and deleted lines thank you! :)

The savings are 2-3 lines per macro; this is a nice improvement to readability. See Technologicat/unpythonic@31106ee

One thing that should probably be documented: args collects positional args given to the call to .transform (well, just like an *args usually does!). When you call .transform, if you want to use gen_sym et al., they should be passed by name to avoid them being inadvertently being picked up by args.

Example. I was using the pattern:

@macros.expr
def mac1(tree, args, gen_sym, **kw):
    return _mac1(tree, args, gen_sym)
def _mac1(tree, args, gen_sym):
    newtree = ...
    return _mac2(newtree, args, gen_sym)

@macros.expr
def mac2(tree, args, gen_sym, **kw):
    return _mac2(tree, args, gen_sym)
def _mac2(tree, arg, gen_sym):
    ... # <implementation goes here>

With b7fd7c4, this becomes shorter, but requires some care:

@macros.expr
def mac1(tree, args, gen_sym, **kw):
    newtree = ...
    return mac2.transform(newtree, *args, gen_sym=gen_sym)

@macros.expr
def mac2(tree, args, gen_sym, **kw):
    ... # <implementation goes here>

Note the differences in the internal invocation of mac2.

It also seems this solution requires some additional calls to ast.copy_location, whereas manually splitting and calling the syntax transformer functions didn't - but it may be that they should be required and the previous implementation didn't just catch that. (Specifically, my aif and do macros now require copying the location information for the expansion to work. In a way it makes sense; they're creating new nodes programmatically.)

[azazel75]

Wow. unpythonic.syntax and its 25 now obsoleted and deleted lines thank you! :)

The savings are 2-3 lines per macro; this is a nice improvement to readability. See Technologicat/unpythonic@31106ee

One thing that should probably be documented: args collects positional args given to the call to .transform (well, just like an *args usually does!). When you call .transform, if you want to use gen_sym et al., they should be passed by name to avoid them being inadvertently being picked up by args.

Yes, I'm still thinking about *args and **kwargs.... do not take that signature as written in the stone ;-)

Example. I was using the pattern:
[...]
With b7fd7c4, this becomes shorter, but requires some care:

@macros.expr
def mac1(tree, args, gen_sym, **kw):
newtree = ...
return mac2.transform(newtree, *args, gen_sym=gen_sym)

you can stop passing gen_sym around, it's added automatically to the call by the machinery

[...]
It also seems this solution requires some additional calls to ast.copy_location, whereas manually splitting and calling the syntax transformer functions didn't - but it may be that they should be required and the previous implementation didn't just catch that. (Specifically, my aif and do macros now require copying the location information for the expansion to work. In a way it makes sense; they're creating new nodes programmatically.)

That's because there's a point where location infos of the AST node are required. This problem surfaced now because the invocation with .transform() happens before the post-execution filters on the calling macro where there is one that fixes missing location infos. It may be a good idea to call that post filter before executing the inner stages of .transform()

[Technologicat]

Yes, I'm still thinking about *args and **kwargs.... do not take that signature as written in the stone ;-)

I think this solution is pretty good, but if you think something else is better, by all means :)

you can stop passing gen_sym around, it's added automatically to the call by the machinery

Ah, thanks! That gets rid of some boilerplate nicely. Technologicat/unpythonic@5914c92

That's because there's a point where location infos of the AST node are required. This problem surfaced now because the invocation with .transform() happens before the post-execution filters on the calling macro where there is one that fixes missing location infos. It may be a good idea to call that post filter before executing the inner stages of .transform()

Ok. Sounds good, especially if it's reasonably possible to make it automatic.

(And if not, I'd be fine with a function that I could call before handing over a tree to a .transform.)

[Technologicat]

Just commenting, this is really useful, I have been using .transform all over unpythonic.syntax.

Even better would be if it was possible for a macro to just expand into another macro invocation (currently doesn't work, at least if both macros are defined in the same module), but this is already very good.

Any news on the API?