Linking tokens and metadata with the same namespace

[amyisard]

In the corpus I am working with we have 2 participants in a signed dialogue, and they each have multiple annotation tiers. We use namespaces in the annotations e.g.

PersonA::Gloss
PersonA::Mundbild
PersonB::Gloss
PersonB::Mundbild

and also in some of the metadata e.g.

PersonA::AltersGruppe

I created the relAnnis corpus using pepper and put in links between tokens from different tiers belonging to the same participant, so the query

Gloss=/ZUG.*/ ->ident Mundbild=/.*zug.*/

will find Gloss - Mundbild pairs from both PersonA and PersonB but only where the Gloss and the Mundbild belong to the same participant.

Is there any way to do this for the metadata?

At the moment if I want to find tokens in the Gloss tier produced by participants in a certain age group I have to use the query

(PersonA:Gloss=‎/KANN.*‎/ @* PersonA:AltersGruppe=‎"31-45‎") | (PersonB:Gloss=‎/KANN.*‎/ @* PersonB:AltersGruppe=‎"31-45‎")

because

Gloss=/KANN.*/ @* AltersGruppe="31-45"

returns results where either one of the participants is in the age group, not just the one who signed the Gloss.

If I have missed that there is already a way to restrict queries to tokens or metadata which are all in the same namespace, please let me know!

[thomaskrause]

There are several answers or possible solutions to this problem caused by the different data models involved when creating the corpus and the limitations/capabilities of these data models. Like you did with the ->ident pointing relations, you need some way of connecting the annotation nodes to the speaker metadata. I don't think it is possible with the pure relANNIS data model (which is used when converting to ANNIS with Pepper), but there might be a way of leveraging the flexibility of the graphANNIS data model with some smaller scripting.

---

In the graphANNIS data model, this could be achieved by adding the meta annotations to corpus/objects for each speaker and then adding PartOf relations to the specific annotation nodes. There could e.g. be three subcorpus nodes and annotation nodes are either connected to the speaker corpus nodes or if they are more generic like the timeline tokens they could be attached directly to the document subcorpus node.

          document
       /            \
   person1        person2
 /    |    \       /    \
 |    |     \     /      \
w1 -> w2 -> w3   w10 -> w11

You could then search something like Gloss=/KANN.*/ @* AltersGruppe="31-45" and only the words that are part of a speaker sub-document that actually have this metadata would be found.
Because a token/annotation node can be part of several sub-corpora and annotations can exist between any node regardless of its PartOf relations, this would still allow grouping them into documents and display them together in the result view.

The obvious problem with this approach is, that no converters exists for directly converting data to the graphANNIS data model (e.g. in the GraphAML format that can be read by the import CLI. In theory, it would be possible to use the Python API to construct such corpora on your own, but this would be a lot of work.

---

By using Pepper to convert the data to relANNIS, two data models are involved: Salt and the relANNIS model.
The latter one is more restrictive, so I would focus on how to achieve something like you want in this data model (in theory one could write a GraphML exporter for Pepper, so one would only be limited by the Salt data model, but such an exporter does not exist yet).

RelANNIS has the concept of documents and texts. A document can consist of several texts, and it is possible to have pointing relations between annotation nodes of any text inside the same document. That could help in your case, but I assume you have some kind of timeline and thus means the relANNIS exporter will create a single text with empty token for the timeline items. Also, in relANNIS only documents can have metadata, not texts. This means you can't use the same structure as outlined for graphANNIS above.

I thought that it might be possible to solve this by adding node annotation labels with the speaker id and than do an "identical value" (==) operator search, but you still only have one document node for several speakers and thus you can't link the annotation to the distinct sub-corpus node belonging only to this speaker.

---

There might be a solution that makes use of the graphANNIS data model, but does not need the implementation of a full conversion script in Python. Instead, you might convert and import the corpus with Pepper and the graphANNIS CLI. Then, you could use the Python API to add some additional annotations and nodes as a post-processing step.

For every speaker you know (probably defined by the namespace of the document node annotations), you create a new graphANNIS node with the node type "speaker". It is important that this node type is neither "node" (an annotation node) nor "corpus" (a document). If this would be "corpus", it would be treated by the ANNIS frontend as document, and it won't be able to handle that there are several documents for an annotation node. Add all relevant metadata like age group to the speaker node. Then find all annotation nodes/Gloss/token that belong to that speaker and add a PartOf relation between the annotation node an the speaker node. It is totally OK for an annotation node to have multiple incoming PartOf edges.

   document ---+
  /   |   \  \  \
 /    |    \  \  \   
 |    |     \  \  \      
w1 -> w2 -> w3 /   \
  \   |   /  w4 -> w5 
    spk1-+    \    /
               spk2

This would allow you to search for

w1 @* spk1

and

w1 @* document

If you want to distribute the post-processed corpus, you can export the modified corpus as GraphML using the graphANNIS command line. Or you just use a script that imports the corpus to a temporary data folder, post-processes it and then exports the result as GraphML.

---

I know that it might be frustrating to not have a simple way of solving this with relANNIS and the Pepper ecosystem. We have some ideas of a graphANNIS based Pepper to avoid the limitations of the Salt and relANNIS data model, but this is more experimental and not an actual project with estimated deadlines.