Error generating TDE

[matt-turner]

Hi - I'm creating a demo using ES on standards documents with a schema called STS. I've got the description up and working, can generate envelop documents, but when I go to validate and install the TDE I get this error:

[javascript] TDE-INVALIDTEMPLATE: (err:FOER0000) tde.templateInsert( -- Invalid TDE template: XDMP-NOTSIMPLE: Node does not have simple content: fn:doc("/source/models/sts-templ.xml")/tde:template/tde:templates/tde:template[2]/tde:templates/tde:template/tde:rows/tde:row/tde:columns/tde:column[2]/tde:scalar-type

I'm attaching my model and the generated template (as a .zip)

There is one array in the definition that defined entity for normative references attached to each standard.

The TDE seems to be stopping on line 132 where there is an empty scalar type. Not sure what should be there to accomplish the definition of the normative refs.

I created all this with EA4 (9.020170113) but just ran it all again with 9.0-20170418. I'm running Mac OS.

Thanks in advance,

Matt

sts-demo.zip

[kcoleman-marklogic]

Charles is out of town, so I'll take a stab. This is not a valid array item type definition.

"norm-refs": {
  "datatype": "array", 
  "items": {"std-ref": {"datatype": "string"}}
}

It should be like this:

"items": {"datatype": "string"}

If you want it to be a reference to some external entity type, then you could set the datatype value to "iri". If std-ref was an entity type defined within this model, then you could reference it like this:

"items": { "$ref": "#/definitions/std-ref" }

Basically, the value of "items" has to have one of the forms you see here: http://docs-ea.marklogic.com/guide/entity-services/models#id_13962.

I am curious:

Did you not also run into problems with dashes in your property names? With my build, which admittedly is from 4/15, tde:validate rejected all the prop names containing dashes. I had to change them all to underscores to get at your scalar type problem.

[jmakeig]

Did you not also run into problems with dashes in your property names?

Yes, he did. Where are the rules for column names documented? Entity Services should generate templates that aren’t invalid by default.

[matt-turner]

Oops - I forgot to mention that part as well. Yes I had to change all the names to conform to SQL column heading standards.

Not sure the best way to resolve that ... but that is also an issue.

I don't think I understand the proposed solution ... so let me just go with what I'm trying to do:

Each standard has a set of references called normative references.

I want to create, in the envelop document, the following pattern:

<norm-ref>
  <std-ref>XXX</std-ref>
  <std-ref>YYY</std-ref>
</norm-ref>

I also want these to define a separate 'table' in the SQL view.

So that's why I tried to use the array construct - but obviously my version isn't correct - with the current definition I was able to create envelope documents but now that I look at them they aren't correctly in an array.

<Standard>
  <urn>iso:std:iso:1161:ed-5:v1:en</urn>
  <doc-number>1161</doc-number>
  <title>Series 1 freight containers — Corner and intermediate fittings — Specifications</title>
  <doc-type>IS</doc-type>
  <originator>ISO</originator>
  <secretariat>AFNOR</secretariat>
  <pub-date>2016-07-15</pub-date>
  <release-date>2016-07-15</release-date>
  <norm-refs datatype="array">ISO 1496-1</norm-refs>
  <norm-refs datatype="array">ISO 148-1</norm-refs>
</Standard>

So how should I properly use the array type?

Thanks,

Matt

[kcoleman-marklogic]

From what you've said, norm-refs represents another entity type. You should model it accordingly. Entity properties can only have 3 possible types: Scalar, array, and reference to another entity type.

[kcoleman-marklogic]

Where are the rules for column names documented? Entity Services should generate templates that aren’t invalid by default.

Since the ES spec doesn't mention this restriction, I was unaware of it until Matt reported his problem. Ergo, it's not documented anywhere, atm. :) However, rest assured that I am going to add it.

@bsrikan recently made me aware of a similar restriction on the model title, imposed by TDE.

I believe the TDE documentation covers this in that it states blah blah must be a valid SQL view name, but that's much too far a distance for a user to bridge. The ES documentation needs to cover it.

[jmakeig]

Entity properties can only have 3 possible types: Scalar, array, and reference to another entity type.

…today. We’re working on how to model “weak” entities, i.e. that aren’t free-standing, but represent some hierarchy under the main entity.

[jmakeig]

The ES documentation needs to cover it.

Shouldn’t we just transform the names in the TDE generation so that they’re valid, such as converting - to _?

[kcoleman-marklogic]

Recall that Charles is out of town. For all practical purposes, that makes this WAI, and it should be documented. Perhaps, an RFE for the next release.

[matt-turner]

So can you guys give me pointers on how to create the proper model? Maybe give me the right way to do this? Thanks, Matt

[kcoleman-marklogic]

I don't think you'll get exactly the structure you want using ES (unless, I suppose, you do something custom in your instance converter code), but a combination of arrays and perhaps a second entity type should get close.

Justin can probably speak more wisely to this than I can, but isn't starting with the entity structure and trying to force the model to conform a bit of the tail wagging the dog?

In any case, I don't at all mean to blow you off, but I'm under a serious deadline crunch right now, so I don't have time to devote to this. Esp. since my expertise is not much beyond your own (if that). Perhaps Justin has time to chime in with better suggestions.

[bsrikan]

Shouldn’t we just transform the names in the TDE generation so that they’re valid, such as converting - to _?

It would be a good idea to rather add this rule to model-validate() so folks dont hit this issue in later stages.

@matt-turner
You can try editing norm-refs as below:

"norm-refs": {
"datatype": "array", 
"items": {
"$ref": "#/definitions/Std_ref"
}
}

and adding another entity type as so:

"Std_ref": {
"properties": {
"std_ref": {
"datatype": "string"
}
}
}

With this you have to edit the generated conversion-module to add a "/*" as shown in Post Edit:
OOTB:
=>es:optional('norm-refs', es:extract-array($source-node/norm-refs, standard:extract-instance-Std_ref#1))
Post Edit:
=>es:optional('norm-refs', es:extract-array($source-node/norm-refs/*, standard:extract-instance-Std_ref#1))

[matt-turner]

Thanks - I'll give all that a try and report back (in a day or so).