Consider alternatives to Stork

[baseplate-admin]

Hi, from today Stork's development has been ceased. For more information please see jameslittle230/stork#360 (comment)

We should find alternative way to handle search.

Some of the alternatives that we can look into:

• Tinysearch
• PageFind
• Lunr
• LyraJS

[justinmayer]

Given that Stork still works very well and doesn't seem to be going away any time soon, I don't think this is a significant issue. I certainly don't believe Stork needs to be replaced in the near-term. That said, I would welcome PRs that add alternative search functionality as options — i.e., in addition to the existing Stork functionality. I would be particularly amenable to support for tools that are philosophically similar to Stork, such as Tinysearch and PageFind.

[Kristinita]

Given that Stork still works very well and doesn't seem to be going away any time soon, I don't think this is a significant issue. I certainly don't believe Stork needs to be replaced in the near-term.

Type: Counterarguments 💬

My Stork problems:

1. #361 — if I use my search-index.st index file 24.2 megabytes in size, Stork takes more than 10 seconds to load on iOS and blocks other elements from displaying on the webpage. Another user writes, that on iOS Stork freezes the page, preventing users from scrolling or interacting with it for a while. This is a critical problem that caused me to remove Stork from my site.
2. #313 — I can’t compile Stork on Travis CI, AppVeyor CI and Circle CI Windows environments. The author of Stork writes that the same problem exists on GitHub Actions. Because of this, I compile the search-index.st index file on my local machine and upload it to the GitHub repository of my static site. This is not what the build process should look like.
3. #316 — Stork shows small number of results in the SERP by default. If we set a large value in displayed_results_count, then site visitors will have performance problems. Stork doesn’t solve the problem of how users can view dozens or hundreds of SERP results without big performance problems.
4. I don’t think using software that is no longer in development is ever a good solution because if any problem arises, no one will fix them.

Stork has critical problems for me. And, of course, I would like to see alternatives to it for use in Pelican.

Thanks.

[justinmayer]

Hi @Kristinita. While I don't personally experience any of the issues you mentioned, I understand that they are affecting you, and I understand your desire to resolve them.

To reiterate what I said above, I would welcome a pull request that implements support for an alternative search tool, particularly Pagefind since it currently seems to be the most similar and the most actively developed.

[danj-ca]

Proposal for adding PageFind to Pelican Search plugin

I was taking a look at PageFind's documentation, and thought I might take a run at integrating it (the search engine, not its documentation 😏) into Pelican. An alternative to Stork appeals to me both because the project is no longer maintained, and because it imposes Rust's Cargo package manager on my site's toolchain—I'd rather just have one type of dependency management (i.e. Python) involved in building my site, if possible!

To make the pelican-search plugin support multiple search engines instead of just Stork, it's useful to be very explicit about what this plugin does / doesn't do¹.

These search engines can be thought of as consisting of two pieces:

1. A tool that generates a search index over the contents of your site.
2. A component to embed in your site to provide a user experience for search, consuming the index.

The plugin's README makes it clear it only addresses the first of those:

This plugin generates an index for searching content on a Pelican-powered site.

So it's important to remember, as we go forward, that switching search implementations on a Pelican site may not be as simple as just adjusting a SEARCH_PROVIDER setting from stork to pagefind; you will also need front-end changes (probably in the site's theme) to provide the correct user experience for the index.

The above may seem to go without saying, but it helps us keep the scope dialled in correctly: this plugin may include instructions for how to make your theme compatible with each search provider, but the plugin itself just supplies a generator that creates a search index.

Design of the Plugin

The plugin currently consists of the SearchSettingsGenerator class², which writes some Stork-related settings from our Pelican config to a TOML file, and then builds the search index by calling out to stork build as a subprocess.

PageFind can also be configured through config files, so we could generalize the Search Generator class to perform these steps:

1. Accept search-provider-related settings from Pelican config
2. Write the settings to the provider's config file
3. Run the provider's index creation tool to produce the index

Generalizing the Structure

I'm not an expert at the Pythonic way to plug and play "settings writers" and "index creators", but I imagine pulling the contents of the generate_stork_settings and build_search_index methods out into search-provider-specific classes, e.g. StorkSettingsWriter, PageFindSettingsWriter, StorkRunner, PageFindRunner, and choosing the desired classes to create when the plugin runs based on a SEARCH_PROVIDER setting in Pelican config.

In Closing

I've sketched out a design for how we could generalize the Pelican Search plugin to support multiple providers. Given the time and attention, I may take a stab at implementing this, as I'd like to try out PageFind on a Pelican site. But I wanted to document the thought process for discussion (please point out what I've missed / differences of opinion!) and in case anyone else wants to take a swing at it.

Footnotes

1. Don't do what Donny Don't does... they could have made this clearer ↩

2. I can't resist pointing out that this class name buries the lede: the important thing it does is generate a search index. The fact it needs to generate "search settings" in order to do that is really secondary. 😛 ↩