poise redesign

[oubiwann]

Problem Statement

The LFE website has been using Zola for many years, and while this has been great, there are things that we would change if we could.

Item 1 - Markdown AST

we'd like to make some significant changes to how Markdown is displayed in certain parts of the site and we don't have the level of flexibility we would like. For instance, taking the headings of a certain level and all their children and rendering those in different parts of a template. To do that more simply/generally, one would like to have access to the AST of the markdown and then be able to call useful Zola or Tera template functions/built-ins on the AST (or select parts of it).

Item 2 - Pages vs. Sections

The page/section distinction that Zola makes is a little bit arbitrary and mostly useful for Zola internals (it's able to do some clever parsing/rendering parallelisation due to this abstraction). However, the downside to the user is a fair bit of confusion and a lot of unnecessary code/template duplication.

Item 3 - Template Layout Complexity

We've never really had a clean Zola template setup (possibly the root cause being "Item 2" above) and we're not really sure how to fix that without spending a great deal of time on some cleanup can can't really justify the time expenditure when what we have seems to work (if it's awkward to maintain/edit).

Item 4 - Zola and/or Tera Extensibility

A nicely extensible system would allow for us to make the above changes to it and still use the core of said system. Neither Zola nor its chosen templating system Tera offer the degree of customisation we need to accomplish our aims.

Item 5 - Clean Multi-site Capability/Workflow

This is probably possible in Zola, we just never were able to figure out a good answer in the various times we tried investigating the matter. Essentially, we'd like to be able to statically generate completely different content, templates, look-and-feel, etc., under one domain (under arbitrary paths). This would allow us to support, e.g., a "main" site, a blog, language documentation, etc.

Proposal

Direction Change

This redsign effort intentionally moves away from the inspiration provided by the Clojure-based stasis project, and instead towards something more Zola-like, but with greater simplicity and more data.

Features

• Site generation:
  • driven by configuration
  • multi-site capable
• Page data:
  • markdown as source, including "front matter" (metadata)
  • parsed as HTML and AST (mdast)
• Template system:
  • erlydtl (this gives us template composition, etc., as well as custom tags/extensibility)
  • clear means of template selection in md files (e.g., front matter)
  • consistent set of variables every page has access to

Weaknesses

This approach has the following difficulties than we need to be aware of (only some of which may be surmountable):

• LFE/Erlang are not as fast as compiled Rust, so generating a site will take longer (large sites potentially much longer)
• Erlang does not seem to have a markdown parser that can generate mdast, so we will likely have to install and use third-party tooling via the CLI (or port calls to the OS)
• As such, we will in all likelihood have to essentially create an in-memory "db" for each file consisting of filesystem path, front matter metadata, raw markdown, markdown AST, and parsed HTML (these + configuration data will need to be passed to the compiled Erlydtl templates)
• Erlydtl compiles templates to Erlang bytecode, which may then have their renderfunction called to output file data ... this means there will be a compile for the templates and any custom Erlydtl tags, a compile for poise code, and then a "run" step that will need to call this code and all associated/required renderfunctions.

Work

• Define generation flow/lifecycle #15
• Core
  • Convert library to OTP app (gen_server)
  • Define top-level configuration
  • Establish convention for template directory layout
  • Establish convention for markdown dirs -> URL paths
  • Define page metadata
  • Devise a means of discovering a page's template if not defined (ancestor metadata search)
  • Define page variables
• Parsing
  • Select markdown tooling #11
  • Devise the means by which poise will utilise md file tooling #12
  • Determine the data format parsed data will be shared with poise #13
• Data
  • Select a means by which data will be stored and accessed for each page #14
  • Devise a method of pulling in all page ancestor data without having to hold an entire site's data in-memory
  • Ensure that all data (configuration, page metadata, md-related data, etc.) are ready to be passed to Erlydtl templates
• Output
  • Establish the convention for ensuring all templates have their render functions called
  • Ensure that project's that have poise as a dependency can use it to easily statically generate content
• Development
  • Investigate notify support
  • Set up a dev server
  • Set up a watcher