Transformer matches can be too conservative #36
Description
Original comment from @EFord36
[kazu/steps/ner/opsin.py extendString() method reworks entity matches to account for Transformer model matches that tend to identify only a part of entities with longer names - which perhaps indicates need for more generic logic around handling transformer model matches]
@RichJackson my gut says maybe this logic should be wrapped up in the TransformersModelForToeknClassificationNerStep like we have the NonContiguousEntitySplitter, so we don't get the match wrong to start with, and need to then fix it here much later in the pipeline. It would also mean either having opsin used to some extent by all users, or having it be pretty flexibly configurable, or writing our own logic to decide when extending is reasonable.
What do you think?