Spurious characters in titles introduced in v1.0.2

[justinmayer]

When testing the code for an upcoming release of the Search plugin, I upgraded from 1.0.1 to 1.0.2 on my local workstation and immediately noticed some spurious characters between the last two words of all the titles in the search results. (See attached screenshot.) I assume those spurious characters are associated with the   added by Typogrify’s widont feature.

So it seems that the json.dumps() method introduced via cc48d48 in #23 was sadly not without its side effects after all. That being the case, I intend to revert that change in the upcoming release and, for the moment, go back to the original fix provided in #15.

@maphew / @s3lph / @lioman: If any of you would like to look at how the json.dumps() method could be re-deployed in this plugin in a way that does not produce these spurious characters, that assistance would be most welcome and appreciated.

[justinmayer]

As noted above, I temporarily reverted to the title-cleaning behavior as existed in #15 in the just-released version https://github.com/pelican-plugins/search/releases/tag/1.1.0. I look forward to any thoughts regarding if and how the json.dumps() method might be re-introduced.

[lioman]

We can strip all non-printable chars by using unicodedata.normalize("NFKD", title) but I'm not sure if we have some side effects there

[justinmayer]

Good suggestion, @lioman. Thanks to the work you did to add some tests for this plugin, this would be a good opportunity to add more tests to ensure this situation is handled correctly.