Unable to redefine recursively included definitions.

[takesson]

Throws error when the original definition does not exist in the immediately included rng.

Included grammar contains no define(s) named ... to replace.

I will attach an example which can be added to test coverage.

[takesson]

include-recursive.zip

[h4l]

Thanks for reporting this & including a reproduction example! I'll have a look into this when I get a chance.

[h4l]

I just had a look into this. I can reproduce what you see, and it is indeed a bug. The problem is caused by the DeferredXmlInsertion workaround to prevent lxml rewriting namespace prefixes.

_remove_overridden_components() is where the error occurs. It's trying to remove a <define name="content"> so that it can override it with the one in start.rng. It fails to find it, because it only looks in intermediate.rng, whereas it actually exists in common.rng. It only looks in intermediate.rng because of the deferred insertion workaround — the deferred insertions are not applied until the end of the whole inclusion process, so indirect inclusions are not available to _remove_overridden_components().

I can see two options for fixing this right now:

1. Eagerly apply the inclusions, rather than deferring them to the end. This will make the indirectly-included elements visible to _remove_overridden_components(), at the cost of exponential growth in cost of applying the DeferredXmlInsertion workaround as the depth of inclusions increases.
2. When searching for elements to remove in _remove_overridden_components(), rather than only looking in the immediate included XML doc, it could descend into the DeferredXmlInsertion tree to find references in yet-to-be-merged docs. This would avoid the exponential cost of eagerly merging inclusions, but I need to look at the implications carefully to check if this could have side-effects that would change semantics of the included content.

So I think 2 is the best option if it works...

I've also got to tidy up and modernise the project tooling a bit, as it's very out of date! I'll do that before applying a fix, so this will take a bit longer than just patching this by itself.

[h4l]

@takesson I just released version 1.0.0 on PyPI which should have fixed this issue, It would be great if you could give it a try and let me know if it's working now.

[takesson]

Thanks, this seems to be working! I have tested with a tougher dataset.

There is a difference btw rnginline and jing simplification with regards to the datatypeLibrary. I am not sure if it is a problem in any way or if it changed but it is mentioned in the code.

Jing simplification:

              <attribute name="scope" ns="">
                <choice>
                  <value type="token" datatypeLibrary="" ns="">external</value>
                  <value type="token" datatypeLibrary="" ns="">local</value>
                  <value type="token" datatypeLibrary="" ns="">peer</value>
                  <value type="token" datatypeLibrary="" ns="">-dita-use-conref-target</value>
                </choice>
              </attribute>

rnginline:

          <attribute name="scope">
            <choice>
              <value datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes">external</value>
              <value datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes">local</value>
              <value datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes">peer</value>
              <value datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes">-dita-use-conref-target</value>
            </choice>
          </attribute>

[h4l]

Excellent, glad it's working! Thanks for testing.

With the datatypeLibrary, I would have to guess that the <value> elements had those datatypeLibrary attributes in the original file. rnginline does have to change some datatypeLibrary attributes, but only on the <grammar> elements that are included. It shouldn't change the value on other elements, like these <value>s. And when it changes them, it only sets the datatypeLibrary value to "" (this is to prevent the included grammar inheriting a datatypeLibrary defined in the including file).

But I don't immediately understand why Jing simplified them to "" when they have a value in the rnginline version. The simplification rules say that it datatypeLibrary would be come "" when there's no datatypeLibrary applied to the element (either from an ancestor, or itself). I wonder if Jing is using the XMLSchema-datatypes URI as an implied default somehow? I expect it must be behaving correctly!

If you could share the full files I'd be happy to take a look.

[h4l]

Ah I see. The spec says:

4.4. type attribute of value element
For any value element that does not have a type attribute, a type attribute is added with value token and the value of the datatypeLibrary attribute is changed to the empty string.

This explains why Jing changed the datatypeLibrary attribute. Basically the value elements had a datatypeLibrary attribute, but no type attribute, so the datatypeLibrary was not having any effect. Jing normalises this by making the type="token" explicit, and removing the unnecessary datatypeLibrary, but rnginline just leaves it as it was.

[takesson]

Interestingly, there is no datatypeLibrary on the value elements in the source. Here is an updated version of intermediate.rng (demonstrates the effect in combination with the original files in include-recursive.zip):

<?xml version="1.0" encoding="UTF-8"?>
<grammar 
    xmlns="http://relaxng.org/ns/structure/1.0"
    xmlns:a="http://relaxng.org/ns/compatibility/annotations/1.0"
    datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes">

    <include href="common.rng">
        <!-- Important: no redefine of 'content'. -->
        
        <!-- Redefine atts in intermediate.rng to demonstrate both levels of redefine. -->
        <define name="atts">
            <attribute name="a"/>
            <attribute name="scope">
                <choice>
                    <value>external</value>
                    <value>local</value>
                    <value>peer</value>
                    <value>-dita-use-conref-target</value>
                </choice>
            </attribute>
        </define>
    </include>
    
    <define name="intermediate.element">
        <element name="intermediate">
            <empty/>
        </element>
    </define>
    
</grammar>

[takesson]

To further improve the test coverage, here is intermediate.rng with an additional attribute that actually gets the datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes" when simplified by jing.

<?xml version="1.0" encoding="UTF-8"?>
<grammar 
    xmlns="http://relaxng.org/ns/structure/1.0"
    xmlns:a="http://relaxng.org/ns/compatibility/annotations/1.0"
    datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes">

    <include href="common.rng">
        <!-- Important: no redefine of 'content'. -->
        
        <!-- Redefine atts in intermediate.rng to demonstrate both levels of redefine. -->
        <define name="atts">
            <attribute name="a"/>
            <attribute name="scope">
                <choice>
                    <value>external</value>
                    <value>local</value>
                    <value>peer</value>
                    <value>-dita-use-conref-target</value>
                </choice>
            </attribute>
            <attribute name="id">
                <data type="NMTOKEN"/>
            </attribute>
        </define>
    </include>
    
    <define name="intermediate.element">
        <element name="intermediate">
            <empty/>
        </element>
    </define>
    
</grammar>

jing:

              <attribute name="id" ns="">
                <data type="NMTOKEN" datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes"/>
              </attribute>