(PDF) Failure to compile difan document due to "invalid XML"

[ronaldtse]

Somewhere in this line it contains a character that kills the PDF generation process. If I remove this line, it does not fail.

天子所命者農，命官所先者農，朝廷所加恵者農，官府所分理者農。下至州郷閭巷，婦人女子，賤𨽻小夫，歌詠讃頌亦無一不在農，噫盛矣！

Failure trace:

org.xml.sax.SAXParseException; lineNumber: 13; columnNumber: 20662; An invalid XML character (Unicode: 0xd863) was found in the element content of the document.
javax.xml.transform.TransformerException: org.xml.sax.SAXParseException; lineNumber: 13; columnNumber: 20662; An invalid XML character (Unicode: 0xd863) was found in the element content of the document.
	at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:536)
	at org.metanorma.fop.PDFGenerator.runFOP(PDFGenerator.java:663)
	at org.metanorma.fop.PDFGenerator.convertmn2pdf(PDFGenerator.java:489)
	at org.metanorma.fop.PDFGenerator.process(PDFGenerator.java:320)
	at org.metanorma.fop.mn2pdf.main(mn2pdf.java:338)
Caused by: org.xml.sax.SAXParseException; lineNumber: 13; columnNumber: 20662; An invalid XML character (Unicode: 0xd863) was found in the element content of the document.
	at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
	at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1465)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.scanSurrogates(XMLScanner.java:1435)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2851)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:542)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:889)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:825)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224)
	at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:519)
	... 4 more
---------
org.xml.sax.SAXParseException; lineNumber: 13; columnNumber: 20662; An invalid XML character (Unicode: 0xd863) was found in the element content of the document.
	at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
	at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1465)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.scanSurrogates(XMLScanner.java:1435)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2851)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:542)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:889)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:825)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224)
	at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:519)
	at org.metanorma.fop.PDFGenerator.runFOP(PDFGenerator.java:663)
	at org.metanorma.fop.PDFGenerator.convertmn2pdf(PDFGenerator.java:489)
	at org.metanorma.fop.PDFGenerator.process(PDFGenerator.java:320)
	at org.metanorma.fop.mn2pdf.main(mn2pdf.java:338)
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/mn2pdf-2.11/lib/mn2pdf.rb:64:in `mn2pdf'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/mn2pdf-2.11/lib/mn2pdf.rb:51:in `block in mn2pdf_hash'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/mn2pdf-2.11/lib/mn2pdf.rb:94:in `block in dump_fontist_manifest_locations'
~/.asdf/installs/ruby/3.3.2/lib/ruby/3.3.0/tempfile.rb:447:in `create'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/mn2pdf-2.11/lib/mn2pdf.rb:90:in `dump_fontist_manifest_locations'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/mn2pdf-2.11/lib/mn2pdf.rb:49:in `mn2pdf_hash'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/mn2pdf-2.11/lib/mn2pdf.rb:31:in `convert'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/isodoc-2.12.7/lib/metanorma/output/xslfo.rb:10:in `convert'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/isodoc-2.12.7/lib/isodoc/xslfo_convert.rb:68:in `convert'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/bundler/gems/metanorma-jis-41227cfaac20/lib/metanorma/jis/processor.rb:53:in `output'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/metanorma-2.0.10/lib/metanorma/compile/compile.rb:184:in `process_output_threaded'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/metanorma-2.0.10/lib/metanorma/compile/compile.rb:176:in `block in process_exts1'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/metanorma-2.0.10/lib/metanorma/util/worker_pool.rb:21:in `block (3 levels) in init_thread'
<internal:kernel>:187:in `loop'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/metanorma-2.0.10/lib/metanorma/util/worker_pool.rb:19:in `block (2 levels) in init_thread'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/metanorma-2.0.10/lib/metanorma/util/worker_pool.rb:18:in `catch'
~/.asdf/installs/ruby/3.3.2/lib/ruby/gems/3.3.0/gems/metanorma-2.0.10/lib/metanorma/util/worker_pool.rb:18:in `block in init_thread'

Archive 2.zip

[Intelligent2013]

Very strange issue. XMLSpy shows the XML is well-formed.

I've copied two characters 賤𨽻 (U+8CE4 and U+D863) into the notepad++ (Win text editor).
When I find the string \x{8CE4}, the character highlighted (gray background):

But when I find the string \x{d863}, it shows the message ^ zero length match that point between before the character U+d863:

Word substitute the different fonts when copy-paste the characters:

U+D863 relates to the 'High Surrogates' block (https://www.compart.com/en/unicode/U+d863).

From https://unicode.org/faq/utf_bom.html#utf16-2:

Surrogates are code points from two special ranges of Unicode values, reserved for use as the leading, and trailing values of paired code units in UTF-16. Leading surrogates, also called high surrogates, are encoded from D800(16) to DBFF(16), and trailing surrogates, or low surrogates, from DC00(16) to DFFF(16). They are called surrogates, since they do not represent characters directly, but only as a pair.

The key phrase is since they do not represent characters directly, but only as a pair..
The next character after U+D863 is 5C0F (https://www.compart.com/en/unicode/U+5C0F).
I.e. as I understood, the next character should be from the range U+DC00- U+DFFF.

[Intelligent2013]

The character https://util.unicode.org/UnicodeJsps/character.jsp?a=28F7B should be used.
But when I copy-paste it from the page, U+D863 occurs in the text.
I've tried to put 𨽻 directly into the Presentation XML, but internally it replaces by U+D863 again...

[Intelligent2013]

I'll try the Xalan patch from apache/xalan-j#4.

[Intelligent2013]

The patch is working partially. The PDF generated, but the processing stopped at the problematic character:

XSL-FO:

[Intelligent2013]

𨽻 (https://www.fileformat.info/info/unicode/char/28f7b/index.htm) encoded as:

• adoc: F0A8BDBB - correct
• Presentation XML: F0A8BDBB - correct
• intermediate update_xml_step0.xml and update_xml_step1.xml (UTF-16) F0A8BDBB - correct
• XSL-FO XML (UTF-8) corrupted

I have to check why UTF-8 encoding of XSL-FO is using instead of UTF-16.

[Intelligent2013]

For XSL-FO encoded in UTF-16, the Apache FOP fails with the error ill-formed UTF-16 sequence, contains isolated high surrogate at end of sequence:

Jan 17, 2025 8:23:30 PM FOP startFOP
SEVERE: Exception
org.apache.fop.apps.FOPException: java.lang.IllegalArgumentException: ill-formed UTF-16 sequence, contains isolated high surrogate at end of sequence
javax.xml.transform.TransformerException: java.lang.IllegalArgumentException: ill-formed UTF-16 sequence, contains isolated high surrogate at end of sequence
        at org.apache.fop.cli.InputHandler.transformTo(InputHandler.java:298)
        at org.apache.fop.cli.InputHandler.renderTo(InputHandler.java:117)
        at org.apache.fop.cli.Main.startFOP(Main.java:180)
        at org.apache.fop.cli.Main.startFOP(Main.java:142)
        at org.apache.fop.cli.Main.main(Main.java:211)
Caused by: javax.xml.transform.TransformerException: java.lang.IllegalArgumentException: ill-formed UTF-16 sequence, contains isolated high surrogate at end of sequence
        at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:739)
        at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:345)
        at org.apache.fop.cli.InputHandler.transformTo(InputHandler.java:295)
        ... 4 more
Caused by: java.lang.IllegalArgumentException: ill-formed UTF-16 sequence, contains isolated high surrogate at end of sequence
        at org.apache.fop.fonts.MultiByteFont.mapCharsToGlyphs(MultiByteFont.java:668)
        at org.apache.fop.fonts.MultiByteFont.charSequenceToGlyphSequence(MultiByteFont.java:572)

There is a patch https://issues.apache.org/jira/browse/FOP-2918. I'll try it.

[Intelligent2013]

The patch https://issues.apache.org/jira/browse/FOP-2918 solves the concrete issue describes in the Jira ticket.
This patch applies to the classes org.apache.fop.layoutmgr.inline.TextLayoutManager.java and org.apache.fop.util.CharUtilities.java, but the exception ill-formed UTF-16 sequence, contains isolated high surrogate at end of sequence occurs in the class org.apache.fop.fonts.MultiByteFont.
I need more investigation.

[Intelligent2013]

The problem occurs in these components:

• Xalan stops the processing just before the character 𨽻.
• Apache FOP:
  • the patch https://issues.apache.org/jira/browse/FOP-2918 solves the issue partially, additionally need to update the class Characters to prevent the ArrayOfOutBoundException in the functions lookupOrientationInRanges and lookupOrientationInSingles.
  • the issue ill-formed UTF-16 sequence, contains isolated high surrogate at end of sequence occurs due the missing glyph in the fonts Noto Serif JP, STIX Two Math, Noto Serif, Noto Serif HK, Noto Serif JP, Noto Serif KR, Noto Serif SC, Noto Serif TC. To render the PDF need to add the font MingLiU-ExtB (available in C:/Windows/Fonts/mingliub.ttc) and in fontist (https://github.com/fontist/formulas/blob/ffa7239af974204836af07348948a155a41b6bf3/Formulas/office_preview.yml#L229).

So, need to find the solution for Xalan issue.

[Intelligent2013]

Actually the issue in the XSLT regex for split the character ranges for vertical layout. For some reason, the sequence \uD863\uDF7B process as two different characters. The regex contains the range \u3000-\u9FFF\.