Inline HTML handling is sloppy

[rexxars]

The fact that foo <strong>bar</strong> gets turned into foo <span><strong></span>bar<span></strong></span> is obviously horribly broken. It's tied to the fact that the walker gives <strong> as an inline HTML element, then gives bar as a text node, followed by </strong> as another inline html element. Not 100% how to best handle this yet.

[kadishmal]

Stumbled upon this exact issue in remarkjs/react-markdown#17.

[rexxars]

Because of the way the commonmark parser is built, this one is pretty tricky to solve, unfortunately. Would love some help on this if someone has time.

[kadishmal]

I guessed that this comes from the parser. Any hint where I should look for this issue?

[rexxars]

Basically, when encountering foo <strong>bar</strong>, the parser yields four tokens:

1. Text with the literal foo
2. InlineHtml with the literal <strong>
3. Text with the literal bar
4. InlineHtml with the literal </strong>

Because of how React works, we can't just inject "half a tag" somewhere. If the parser gave us <strong>bar</strong>, things would be a lot simpler, but this doesn't work because of things like the following: foo <strong>*bar*</strong>. In other words, markdown inside of inline HTML.

[kadishmal]

Yeah, I see. Seems like have to fall back to the plain Markdown processor and inject the output dangerously.

[tauren]

I also ran into this issue in remarkjs/react-markdown#67

Is this an issue in the commonmark parser, or the way it is used by commonmark-react-renderer? Has anyone looked into alternative commonmark parsers?

[rexxars]

The issue is basically that you'd need to pull in an HTML parser in order to properly handle this.
Let's imagine you have this HTML:

<span id=unquoted class="foo">Some *bold* <span class='outlined'>text</span></span>

In an ideal world, it would have to convert this to the following tree:

{
  tag: 'span',
  attrs: {id: 'unquoted', className: 'foo'},
  children: [
    'Some ',
    {tag: 'strong', children: ['bold']},
    {tag: 'span', attrs: {className: 'outlined'}, children: ['text']}
  ]
}

Which is non-trivial to implement with a parser that only emits tokens without having any notion of "depth". You'd have to keep track of the structure you're inside of, so that for the opening <span>, you'd note that we're now entering a node and need to record every new token as a child, until you run into a new opening tag or a closing tag for the span. Then you'd also have to consider cases like optional closing tags (good old <p>, for instance), incorrect HTML structures (<span>Foo<em></span>Muh</em>) and the likes. It really is a nightmare.

[tauren]

What are the objections to pulling in an HTML parser? Is it the size? Does the commonmark parser not help with this? Would changing parsers solve this?

The way things are now, inline html doesn't work, even for very simple and properly formed html:

<span>Hello</span>

Here is what gets rendered:

<p><span><span></span></span><!-- react-text: 89 -->Hello<!-- /react-text --><span></span></p>

I suggest that the project README reflect this situation until it can be solved and it should also be mentioned in the react-markdown readme. I suspect this might save others a bunch of time figuring out why things aren't rendering right.

[rexxars]

Well, even if you pull in an HTML parser (which will add significant weight), you'll still have to figure out how to deal with all the edge cases, as I outlined above (broken HTML, optional closing tags etc).

It's not a task I want to venture out on - my time is already stretched way too thin as it is. I'd be more than happy to accept a pull request that would improve the situation. Last time I thought about this problem, I came across html-to-react, which would solve much of this problem if only the commonmark parser returned blocks of HTML instead of just tokens for individual HTML fragments (start and end tags as individual tokens). So you'd still be left with trying to match up start and end tags manually.

I'll try to update the readmes to reflect this state.

[tauren]

Thanks for the explanation, and I understand being stretched too thin. I was hoping to get a clear picture of how you would solve this so that someone else could take on the work and create a PR you would potentially accept. It would also help to know what you don't want to see (adding significant weight to project, etc.).

I am stretched thin and don't have the time to solve this problem either. I'm working around the issue in my current project, but it keeps rearing it's ugly head and I may end up needing to work on a real solution.

[kentcdodds]

Hi there! I'm evaluating this project right now. Honestly, I wouldn't expect the contents of HTML to be parsed as markdown. Couldn't things be drastically simplified if we say that once you get into HTML-land we no longer parse it as markdown? Kinda like switching context from JSX to expressions and back again using { and }?

[rexxars]

Hi Kent!

For sure, things would be simpler if inline HTML was just emitted as one big chunk. I don't think this is consistent with other markdown parsers though, and might not necessarily be what you want. It also doesn't really solve the problem unless we switch to a different parser.

As outlined earlier, the parser itself outputs individual tokens. If it emitted a single token, it would be a different story.