Understand a reasonable workflow for generating new annotation types (gather evidence)

[elmisback]

Steps:

1. Generate a new annotation type and document everything you had to do from start to finish
2. Repeat (1) until you recognize clear patterns
3. Write up documentation of the patterns
4. Ask a language model to come up with an idea for an extension, then to follow the documentation to implement its idea. Iterate on the documentation until a model can reliably implement its ideas.
5. Do (4), but with a human (a friend, someone unfamiliar with the system).

We hypothesize that 1 will not be zero-shot. User will want to tweak outputs, either by re-instructing or manually. Outputs may have errors. If re-instructing, users will probably want model to be aware of the old prompt.

Our goal is to scaffold the process so that users feel like there is a clear workflow for generating a new annotation type. Ideally, even advanced users should not have to start going off-road to get what they want; going off-road should definitely not be normal.

Think of your own ideas for annotation types. Also ask a language model to generate ideas for annotation types.

To test your documentation, try giving the documentation to a language model and having the model work through the process as you understand it. Give the model feedback that a user would see.