Code generator for tests / documentation from additional metadata in the rule?

[toumorokoshi]

Looking at #14, it looks like there's some content that might be duplicated across various places, and could perhaps be simplified (for both the author and future reviewers) using some code generation.

Specifically, I'm thinking:

1. Include additional information in each rule
2. Write a script that will parse the rules, extract the values necessary for documentation and tests, and write boilerplate code.
3. Generate the docs and test via this script.

The additional fields in the rules could look like:

aep-133-id-parameter:
  description: "This rule looks at the parameters of "post" operations on a path that does not have a ":" in its final segment
(indicating a custom operation), and complains if there is no parameter with the name `id`."
  examples:
    incorrect:
      - paths:
        '/test1':
          post:
            description: 'No parameters'
        '/test2':
          post:
           description: 'No id parameter'
           parameters:
              - name: force
                in: query
                schema:
                type: boolean
            responses:
              '200':
                description: Ok
        correct:
            - paths:
                '/book':
                  post:
                    parameters:
                    - name: id
                      in: query
                      schema:
                        type: string

Tests can be generated by reading off incorrect examples, and generating a test to make sure those fail. For correct examples, we could generate a test to make sure those pass.

The "disabling" part of the docs can be completely generated without any additional information (that just includes the rule, as well as the matcher).

thoughts?

[mkistler]

I like this idea but I think I would put the examples into their own files rather than tucked into the ruleset. I've been thinking for a while that the tests are a bit unwieldy, in particular coding the test files in js. Separating the test files out so they can be normal yaml, with a standard test runner, would be a good first step, and then we can follow up with generating the docs from those too.

[toumorokoshi]

Sounds good! so maybe a tests or examples directory, with some way to determine if it's a good or bad example?

something like:

aep/
  0131.yaml
  0131-example/
    aep-131-http-body/
      valid/
         contains-http-body.yaml
       invalid/
          does-not-contain-http-body.yaml
    aep-131-http-body.yaml #or perhaps a single file?