A question around BlockIgnores / getting vale to play nicely with Docusaurus

[pklaschka]

Hey there!

I'm currently trying (unfortunately struggling) to set up vale with a Docusaurus-2-based documentation project (i.e., MDX & Co.).

Most things are working pretty well, but we use some custom MDX Components I'd like to ignore using BlockIgnores, but struggle to achieve the desired result (there seem to be problems as we document a React library-based project, and therefore, MDX can also appear in code blocks, which is when everything turns south 😅 ).

Thank you so much in advance for any help you may be able to provide 👍 🙂 . I've literally been sitting on this problem for over twelve hours now, trying to wrap my head around what's going on (unfortunately, I don't know Go enough to really figure out what's going on) 🙈

The MWE .mdx file

<Image
	src="We can do this"
	alt="we-don't-talk-about-bruno"
/>

```tsx title='widget.tsx' {2-4}
code "with" we

<ABC>
Hello world!
</ABC>
```

We do stuff, therefore we are stuff.

Desired result:

PS C:\Users\pablo\GitHub\telestion-docs-1> vale .\test.mdx

 .\test.mdx
 14:1   warning  Try to avoid using              Google.We
                 first-person plural like 'We'.
 14:14  warning  'therefore' is too wordy.       write-good.TooWordy
 14:24  warning  Try to avoid using              Google.We
                 first-person plural like 'we'.

✖ 0 errors, 3 warnings and 0 suggestions in 1 file.

Situation without BlockIgnores

The src in <Image ... />, which in MDX is part of the syntax (and shouldn't get linted) gets checked and creates an error in the linting result.

.vale.ini

StylesPath = vale-styles

Vocab = Docs

[formats]
mdx = md

[*.{md,mdx}]
BasedOnStyles = Google, Vale, write-good
Google.Colons = No
Google.Headings = No
TokenIgnores = (\"+[^\n$]+\"+)

Terminal Output

PS C:\Users\pablo\GitHub\telestion-docs-1> vale .\test.mdx

 .\test.mdx
 2:2    error    Did you really mean 'src'?      Vale.Spelling
 14:1   warning  Try to avoid using              Google.We
                 first-person plural like 'We'.
 14:14  warning  'therefore' is too wordy.       write-good.TooWordy
 14:24  warning  Try to avoid using              Google.We
                 first-person plural like 'we'.

✖ 1 error, 3 warnings and 0 suggestions in 1 file.

(this would be perfect if 2:2 wasn't an error 😅 )

Situation with BlockIgnores

The MDX gets ignored, as desired. However, the MDX in the code example also is affected and for whatever reason, "destroys" everything further down the file.

.vale.ini

StylesPath = vale-styles

Vocab = Docs

[formats]
mdx = md

[*.{md,mdx}]
BasedOnStyles = Google, Vale, write-good
Google.Colons = No
Google.Headings = No
BlockIgnores = (?s) *(<[^>]*?>)
TokenIgnores = (\"+[^\n$]+\"+)

Terminal Output

PS C:\Users\pablo\GitHub\telestion-docs-1> vale .\test.mdx

 .\test.mdx
 10:11  error  Don't use exclamation points    Google.Exclamation
               in text.

✖ 1 error, 0 warnings and 0 suggestions in 1 file.

I would have expected the MDX in the code section to be affected, as well, but not "destroy" everything after it (given that the code wouldn't be linted anyways, as long as my block ignore doesn't accidentally close the code block, I wouldn't have imagined it to be a problem.

In other words: I would have expected a substitution like the following (apart from my substitution not keeping the correct line numbers) to take place, which would have resulted in the desired linting result: https://regex101.com/r/UgRLZh/2

[jdkato]

Is it possible for you to distinguish between your actual components and those being used as examples by their name? Something like (?s) *(<(?:Image|Another)[^>]*?>) perhaps?

[pklaschka]

Thanks for your quick answer! I thought about something like that, but if there's some way around that, I would like to avoid this (since I don't have a lot of control about component naming in the code examples and stuff like <Image />, which I use in the MDX could realistically also be a part of the code examples + this would be a weird edge case to detect and possibly miss linter errors below that if we miss it somewhere).

I'm curious: How does the "processing pipeline" look like for this? If I could understand what happens with BlockIgnores, I might be able to work on a solution. (on a technical level: what's the input string, and when are the BlockIgnores processed? if it were just the raw file, I don't think I'd run into this issue, since the result from the regex101.com link doesn't have this issue)

In essence (due to my lacking understanding of Go and thus inability to work out what happens behind the scenes), my problem right now is that the BlockIgnores behavior is different from what I would have imagined it to be (as described above, the behavior I would have expected shouldn't break the linter as it does) and the actual behavior is a black box I can't figure out (to be clear: this isn't the project's fault, given that it's open-source. It's just that I'm not good enough at reading Go code, and the "pipeline" unfortunately doesn't have very extensive documentation right now; I'd be happy to submit a PR to the docs once I understand it myself to change that 😜 ).

In other words / tl;dr 😉 :

The following is my high-level expectation of vale's behavior. Could you maybe tell me where I'm going wrong with this (since, if it worked like this, the linter output would be as I would expect it to be, which it unfortunately isn't 😅 ):

Thanks again!

[jdkato]

Vale can't simply remove ignore matches because it would lose information necessary to calculate locations. So, you're going to need to be able to distinguish between your components somehow.

You could use configuration comments to avoid using BlockIgnores altogether or use a pattern like above.

In terms of the "pipeline," consider the substitution to be the equivalent of a code block (block) or code span (inline).

[pklaschka]

@jdkato Thanks once again for your quick answer.

You could use configuration comments to avoid using BlockIgnores altogether or use a pattern like above.

There, Docusuarus crossed my plans (I've already tried it, but for whatever reason, Docusaurus doesn't like HTML-style comments surrounding MDX 😅 ).

Vale can't simply remove ignore matches because it would lose information necessary to calculate locations. So, you're going to need to be able to distinguish between your components somehow. [...] In terms of the "pipeline," consider the substitution to be the equivalent of a code block (block) or code span (inline).

Ok 🤔 . But even so (sorry for being so persistent on that point, but it feels like this is an important concept to know about when using BlockIgnores): I don't get how my BlockIgnores block could interfere with vale the way it does (apparently disallowing it from finding the boundaries of the code block), since it also has nothing to do with backticks (i.e., nothing that should interfere with code block detection).

Even if I turn my mental model around (BlockIgnores after the native code block detection), vale behavior doesn't make sense: There shouldn't be (in my mental model) a way this BlockIgnores could mess up the code block detection.

If I had some code like this, I could understand how this could mess with things:

```
<Tag>
```
</Tag>

But in my example, the pattern is either entirely in a code block or fully outside a code block. In other words: the code block detection, whether it runs before or after it, shouldn't have any problems with it.

The only thing that could make some sense to me (if that feeling were correct, I think this should be documented, as I wouldn't expect it to be true) would be if what gets passed to the BlockIgnores processor was some compiled HTML version of the Markdown, i.e., if code got represented by XML-tag-based Markup, as well, in which case it would make at least some sense that my ignore pattern could mess with the code block detection. But it doesn't feel like this is the case, either 🤔 .

Long story short: I'm baffled about what's going on here / how the code block detection and BlockIgnores can interfere with each other in my example. This misunderstanding, of course, could indicate that more documentation around the behavior would be great; I'd be happy to submit a PR for that if I understood it myself 😅 .

Thanks again for your patience 🙂! Even if I can't find an optimal solution for this specific problem, I would still like to understand what's going on / how to properly use BlockIgnores since I'm a huge fan of vale and the fact that its behavior makes no sense to me bugs me.

[jdkato]

There, Docusuarus crossed my plans (I've already tried it, but for whatever reason, Docusaurus doesn't like HTML-style comments surrounding MDX 😅 ).

Vale's own documentation does exactly what I mentioned: it uses HTML-style comments around blocks that need to be ignored in a Docusuarus-built site.

If you have a lot of these blocks, it will probably feel a little cumbersome but it works.

[pklaschka]

Vale's own documentation does exactly what I mentioned: it uses HTML-style comments around blocks that need to be ignored in a Docusuarus-built site.

If you have a lot of these blocks, it will probably feel a little cumbersome but it works.

Somehow, our Docusaurus doesn't play nicely with Vale ignore comments. So be it, I've found another solution:

tl;dr: The solution

Using four backticks for code blocks instead of three backticks makes the BlockIgnores shown above work as expected (at least as I expected it to work 😉 ).

The explanation

I wanted to find the cause for this issue, so I installed some go-tooling and got the Intellij-Go-IDE (GoLand) to debug the project for me (I'm not good at GoLang, but I can use Intellij-like IDEs 😜 ).

What I saw was that vale "ignores" BlockIgnores statements by wrapping them in code blocks. This, however, caused interference when such blocks were found in an already existing code block:

This:

Became this (with the Ignores, based on the replacement in md.go:31):

Which, in turn, became this HTML that then got linted (by the time it reaches md.go:58):

It's pretty obvious why the output from running over this HTML would be wrong 😉

With four backticks for the code block, this was the resulting HTML, which linted perfectly:

I'm not really sure if this should be considered a bug with vale (it might be possible to pre-process "n >= 3 backticks" to "n + 1 backticks" before the replacements to avoid this around md.go:30?) or is the intended behavior (in which case, I think this should be documented somewhere. I wouldn't have expected BlockIgnores to interfere with code blocks, to be honest).