Enhance AutoIndexer to make it easy to filter files

[TomFreeman]

Our use-case is similar to seen elsewhere.

Create a public / private documentation file out of a single set of shared specifications.

We could do this by manually indexing the "public" file, and using autoIndexer for the "private" one, but that seems laborious.

AutoIndexer doesn't seem to have a good extension point for this, so at the moment the best approach I can see would re-create a significant amount of AutoIndexer just to add the filter step in the middle.

The preferred approach at this point would be to enable users to create a "filteredPathsIndexer" in their own repo like so:

import AutoIndexer from '@/AutoIndexer';

export default function (): string { 
	return AutoIndexer.getIndexYaml(this.env.globals.currentFilePointer, 
		{    
			paths: true,
			filterFunction: (path) => { return true; }
		});}

[TomFreeman]

Or, probably even simpler, just expose the filterFunction as an option on all the classes using AutoIndexer, so you can create the function inline in the template.

But I'd be concerned about the developer experience that way.

[p-mcgowan]

That would be a cool feature for sure, and the use-case makes perfect sense.

I'm not sure exactly how you would pass in a custom filter function (I guess you are running boats programatically, but otherwise it would be parsing the function string from the templates), but something in the templates with variables and filter arguments could work:

{{ 
  autoComponentIndexer({
    include: [...],
    exclude: [
      'SomethingModel',
      '^/admin/'
    ]
  }) 
}}

That being said, it's possible to emulate this behaviour using 2 features already: helpers and variables.

This should work in your case for paths - you can do the same for components and other stuff. You can extend / override the original method(s) from helper files, so you don't have to rewrite everything.

./helpers/filteredPathIndexer.js:

const AutoIndexer = require('boats/build/src/AutoIndexer').default;

module.exports = (njk) => function (params) {
  // if the "include_private_paths" variable was passed from package.json, 
  // ignore excludes and do the default behaviour
  if (this.env.globals.include_private_paths) {
    return AutoIndexer.getIndexYaml(this.env.globals.currentFilePointer, { paths: true });
  }

  const excludes = params?.exclude?.map?.(pattern => new RegExp(pattern));
  // or if you don't want to loop: new RegExp(excludes.join('|')) 

  const oldFiles = AutoIndexer.getFiles;

  // overwrite the file getter, filter the paths by the excludes list
  // You could also build a faster walker if most of your files are exluded
  AutoIndexer.getFiles = function (dir) {
    const filesToBeFiltered = oldFiles.bind(AutoIndexer)(dir);

    return filesToBeFiltered.filter(path => {
      for (const disallow of excludes) {
        if (disallow.test(path)) {
          return false;
        }
      }

      return true;
    });
  }

  const indexYml = AutoIndexer.getIndexYaml(this.env.globals.currentFilePointer, { paths: true });
  AutoIndexer.getFiles = oldFiles;

  return indexYml;
}

and src/paths/index.yml.njk:

{{
  filteredPathIndexer({ exclude: [ '/admin/' ] })
}}

you can use variables in package.json to control when the files are ignored, and pass in the path to the helper from there:

{
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "prebuild": "rm -rf dist",
    "build": "boats --functions ./helpers/filteredPathIndexer.js --input src/index.yml.njk --output dist/out.yml",
    "build-public": "npm run build -- --variables include_private_paths=true"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "boats": "^2.8.0"
  }
}

my test folder structure:

.
├── helpers
│   └── filteredPathIndexer.js
├── package.json
└── src
    ├── definitions
    │   ├── index.yml.njk
    │   └── weather
    │       ├── model.yml.njk
    │       └── post.yml.njk
    ├── index.yml.njk
    ├── parameters
    │   ├── index.yml.njk
    │   └── pathId.yml.njk
    └── paths
        ├── index.yml.njk
        └── v1
            ├── admin
            │   └── get.yml.njk
            └── weather
                ├── {id}
                │   └── get.yml.njk
                └── post.yml.njk

then, npm run build will generate the public version:

...
paths:
  /v1/weather:
    post:
      tags:
        - weather
      summary: weather data
      description: Create a new weather record.
      operationId: v1WeatherPost
      parameters:
        - in: body
          name: v1WeatherPost
          description: Optional description in *Markdown*
          required: true
          schema:
            $ref: '#/definitions/WeatherPost'
      responses:
        '201':
          description: Successful temp creation
          schema:
            $ref: '#/definitions/WeatherModel'
        '422':
          description: Invalid form data provided
...

and npm run build-private will generate the private version:

...
paths:
  /v1/admin: # pew pew secrets
    get:
      tags:
        - admin
      summary: admin list
      operationId: v1AdminGet
      responses:
        '200':
          description: All the passwords
          schema:
            type: array
            items:
              type: string
        '401':
          description: new phone who dis
        '403':
          description: no no no
  /v1/weather:
    post:
      tags:
        - weather
      summary: weather data
      description: Create a new weather record.
      operationId: v1WeatherPost
      parameters:
        - in: body
          name: v1WeatherPost
          description: Optional description in *Markdown*
          required: true
          schema:
            $ref: '#/definitions/WeatherPost'
      responses:
        '201':
          description: Successful temp creation
          schema:
            $ref: '#/definitions/WeatherModel'
        '422':
          description: Invalid form data provided
...

admittedly, it's a fair bit of extra work, but these helpers are really just generic, flexible scripts that give you access to the boats env at runtime - so it's extra work up front, but it means you can pretty much do anything with them

[TomFreeman]

So, it complicates things slightly, but we need to filter out files based on their content, as we have an existing structure that tags content. But that doesn't change too much.

I had originally thought to expose a hook point, that someone could use in a custom helper, effectively just making the extensibility more obvious, but requiring around the same amount of work as your example.

But, from your example, just overriding the behaviour of getFiles is broadly the same in terms of programmer experience, with the main downside that AutoIndexer isn't deliberately exposing it as an extension point, so it could be fragile if the implementation changed.

[p-mcgowan]

That's true - fragility-wise it might make sense to roll your own walker in that case, but the idea there was to reduce the amount of overhead in the helpers, but I think that will be the case even with the template api.

Yeah it changes the story a bit if you want to filter by content. For example, if the built-in helper supported a more complicated api, such as exclude: { tags: ['auth'], security: ['x-auth-token'] } or something to that extent, that would be a more complicated change, and would be fairly dependent on making changes to the underlying AutoIndexer to support any of these additional filters.

I think in this case, the helper style shim is still significantly more flexible, in that any additional functionality on a per-project basis can be stored in the source repo as opposed to baked into the core, mostly just because any functionality that isn't "natively" supported can be "hacked in" in a way.

Using the content filtering, for example, and the previous helper file you could iterate the files, load them into an object (I think it's require('yaml-js') or something like that), then assert that !template.tags.includes('private') during the filtering. You would also have to handle dereferencing in the case of root refs (#/definitions/...), but there is another method where that logic lives. This is still more powerful than an inline solution in that for this use-case you might filter by tag, but someone else might want to filter by security definitions.

I think at the end of the day it's 2 sides of the same coin, but until we add that kind of granular support it is technically possible to get the desired behaviour using the custom helper feature. The more custom helper code you use as opposed to the original helper, the less likely it will break if we make changes to the AutoIndexer, the obvious downside being that it's certainly more overhead up front

[p-mcgowan]

That being said, in this case a filter function would be the most flexible option in terms of adding the api to the parser but, at least at the AutoIndexer level, that would still only support path filtering - for content, it would require building the index, loading the files, and filtering based on that content, probably the same way as above. It could be a very useful feature to many people who may not want to always roll their own helpers. This could also be similarly implemented by hosting a library of helpers somewhere - for example the user could just npm i boats-filtered-indexer-helper and other people could contribute helper files that could augment or extend common use-cases, and the boats core could be more-or-less a slimmed-down version which supports these "plugin-style" helpers.

Either way, I guess there are a few ways to go about it. If we could nail down a template api for this feature, it's also definitely an option to extend the autoindexer, or even just roll another one that supports this filtering so that, as you said, changes to one don't affect the other.