Plugin SPI for pegdown

[jroper]

I'd be very interested in a plugin SPI for pegdown.

My particular use case is this. Play framework currently uses markdown for all its documentation. We'd like to pull all the code samples out of the documentation, and have them in Scala files that get compiled and tested, so as to ensure that our documentation always stays current. The samples can be annotated with comments, and then referenced in the documentation, so when the documentation is compiled, it gets pulled in. Akka does something very similar to this, but using RST. We could move to RST, but that's a huge task, if we could stick with markdown, it would be much nicer. This is probably not a feature that belongs in the core markdown library, but being able to plug it in would be very helpful.

I'll have a go at implementing this, but I'd like to get feedback here as to whether it would be something that would be accepted into pegdown. I think it could be useful for more than just our use case. It may even be a clean way forward to adding new features to the pegdown library itself.

[sirthias]

James,
sure, I'd be completely open to this.
specs2 is also using pegdown in its code (for generating the acceptance docs), so maybe it'd benefit from this addition as well.
Let me know what you need from my side.

[jroper]

My progress so far is that I added some plugin points to Parser, so that you can extend Parser to add support for new block or inline constructs. Similarly you can extend the ToHtmlSerializer to serialize any new node types.

This is enough for my purposes, however it's not that useful for a generic plugin interface, since you couldn't plug in multiple extensions, because you can't do multiple inheritance. So I'm going to look at that next.

As I understand it, the parser is enhanced by parboiled, and this seems to do quite a lot of magic stuff, for example it wraps action code in anonymous classes (right?). So, plugins would also have to be enhanced. I don't know what state is used/shared in the enhanced code though. For example, could I define a Rule in one class, and another Rule in another, enhance both classes, and then compose those two rules together? This is what would be needed for plugins.

So I'll have a go at doing this and see what happens, but if you could give me some pointers in the right direction, that would really help.

[jroper]

Ok, so to answer my own question about using multiple enhanced parsers, it seems that a combination of ContextAware and the enhancement code will magically transfer the state (context) of the parser. So everything is probably fine, right?

With that in mind, I've implemented some support, and attached the code to this issue. I am not at all emotionally attached to the approach, the naming, or anything like that, so please provide feedback, I'm happy to completely change it all if you would prefer another approach.

Since parsing and serialisation are two different things, there are two different plugin mechanisms, one for the parser, and one for the HtmlSerializer. Most plugins would probably implement both, but it is possible that a plugin might just implement the parser plugin interface.

For the parser, there are two plugin points, one for inline plugins (inside a paragraph), and one for block plugins. These are provided to the parser using the PegdownPlugins class. For convenience of use, this comes with its own builder. You can either pass individual rules to this builder (which is what you probably would do if you were using Scala rules), but you can also pass it a parboiled Java parser class which implements either InlineParserPlugin or BlockParserPlugin or both. PegdownPlugins will enhance this parser for you, so as a user of a plugin you just need to pass the class to it (and the arguments for that classes constructor, if any). To implement the plugin, you would write a normal parboiled parser, and implement the appropriate parser plugin interface. You can extend the pegdown parser, this is useful if you want to reuse any of its rules.

For the serializer, I wrote a ToHtmlSerializerPlugin interface. It is called when a node that the ToHtmlSerializer doesn't know how to process is encountered (ie, one produce by a parser plugin). It's accept method is passed the node, the visitor (so if the node contains child nodes, they can be rendered using the parent), and the printer for the plugin to print to. The accept method returns true if it knew how to handle the node, or false if otherwise, and the ToHtmlSerializer loops through each plugin, breaking when it reaches one that returns true, and if it finds none, throws an exception like it used to.

[jroper]

@etorreborre Would the ability to plugin to pegdown be useful to you with specs2? If so, you may want to take a look at this too.

[sirthias]

James,
What's your timeline on this? Does play need this right away for just at some point in the future?

I'm asking also because there is another major pull request open (#72), which provides a full port of pegdown from Java to Scala. This has a number of benefits, from a much better parser readability and type-safety to getting rid of the expensive magic parser extension step required in Java. I could see a plugin SPI taking a much better shape in Scala as well.
If things are not pressing on your side I could see the plugin SPI become part of pegdown V2, which will be written completely in Scala. (We are also working on a (scala-only and macro-based) parboiled V2, with a completely different basic architecture but the same DSL, which will significantly improve parsing speed...)

[jroper]

We'd like to start using it as soon as possible (this week), but there's no reason why we couldn't use our own fork, it's only for rendering Play's docs, not something that gets distributed as part of Play's runtime.

Looking over the Scala rewrite, I think the general approach I've taken may still make sense (MarkdownParserConfiguration would include a list of rules for block and inline plugins, and these would be included at the relevant points, while plugging in to the ToHtmlSerializer would be a list of PartialFunctions or functions that returns Option[String]). If that's the case, then it will be no problem for us to use this approach now, and then port it to the equivalent in Scala when pegdown 2 is released.

So I wouldn't mind if you took a look over it and commented on whether you agree with the approach or not, if you do agree with the approach that's all I need for now. And if you let me know when porting to Scala is complete, then I'm happy to update this pull request to work with that (in an idiomatic Scala way of course).

One other thing, we would be interested in both Scala 2.9.x builds (because we use pegdown from the Play SBT plugin to dynamically render Play docs in the development Play server, this is tied to the SBT Scala version, that is, Scala 2.9.2) and Scala 2.10.x builds (for the Play website itself). I've noticed the Play version is 2.10.x, are you considering cross building to Scala 2.9.x?

[sirthias]

James,

I see. From quickly looking over your approach I think it's fine, so please go ahead and use it for your custom pegdown "fork" until I've had time to properly upgrade pegdown to a Scala-based V2 including an "official" plugin architecture.

I've noticed the Play version is 2.10.x, are you considering cross building to Scala 2.9.x?

I think you meant to ask for the parboiled version, right?
Parboiled is already cross-built for Scala 2.9 and 2.10, so no worries there.
(A new macro-based parboiled V2 would of course not work for 2.9, but this is likely so far out that SBT 0.13 will be out already and Scala 2.9 not an issue any more...)

[etorreborre]

For specs2 I've taken a different approach. This is still work in progress (here's an example) but the idea is the following:

• write a specification with an interpolated string which will be rendered with Pegdown
• the interpolated parts can be "snippets" (with the Snippets trait) which capture code, possibly using comment markers (// 8<---) to delimit only what's interesting
• being some real code inside the specification, it can be compiled and auto-completed from inside the IDE
• the last value of the block can either be displayed, or hidden or checked with a specs2 matcher

I still don't know how practical this is as I need to use it on specs2 User Guide. This approach is risky though as I already faced a compiler error (tag not found) which I couldn't solve.

James you can of course steal all that code if you want to build on it, the most interesting part being the String interpolation (in SpecificationStringContext) where a macro grabs the text corresponding to each expression interpolated in a String.

[jirutka]

@sirthias, this issue can be closed IMHO since the plugins support is already merged into upstream.