Out of memory

[BjornDNA]

Validator v.1.18.1
Testing wtih the -p profile

Running on a batch of files resulted in a error message with below. Seems similar to issue #262
The file in question that generates the error is 9,15 MB. The file is converted from a XLSX to ODS via LibreOffice.
Seems like it is not just a question about filesize, as i have been able to validate files with a size of 140 MB. So rather a question of amount of content in the file - suspicion is alot of number of rows/columns

UPDATE: Looking at 7 files that fails validation they are between 8-53 MB in size and have between 200.000 and 1.000.000 number of rows.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.apache.xerces.dom.DeferredDocumentImpl.getNodeObject(Unknown Source)
at org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Unknown Source)
at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeChildren(Unknown Source)
at org.apache.xerces.dom.ParentNode.hasChildNodes(Unknown Source)
at net.sf.saxon.dom.DOMNodeWrapper.getSuccessorNode(DOMNodeWrapper.java:961)
at net.sf.saxon.dom.DOMNodeWrapper.getSuccessorElement(DOMNodeWrapper.java:939)
at net.sf.saxon.tree.util.SteppingNavigator$FollowingElementStepper.step(SteppingNavigator.java:158)
at net.sf.saxon.tree.util.SteppingNavigator$DescendantAxisIterator.next(SteppingNavigator.java:271)
at net.sf.saxon.tree.util.SteppingNavigator$DescendantAxisIterator.next(SteppingNavigator.java:199)
at net.sf.saxon.value.SequenceExtent.from(SequenceExtent.java:49)
at net.sf.saxon.om.SequenceTool.toGroundedValue(SequenceTool.java:55)
at net.sf.saxon.xpath.XPathExpressionImpl.evaluate(XPathExpressionImpl.java:239)
at com.helger.schematron.pure.xpath.XPathEvaluationHelper.evaluate(XPathEvaluationHelper.java:63)
at com.helger.schematron.pure.xpath.XPathEvaluationHelper.evaluateAsNodeList(XPathEvaluationHelper.java:81)
at com.helger.schematron.pure.bound.xpath.PSXPathBoundSchema._validateSerial(PSXPathBoundSchema.java:696)
at com.helger.schematron.pure.bound.xpath.PSXPathBoundSchema.validate(PSXPathBoundSchema.java:796)
at com.helger.schematron.pure.bound.AbstractPSBoundSchema.validateComplete(AbstractPSBoundSchema.java:259)
at com.helger.schematron.pure.SchematronResourcePure.applySchematronValidationToSVRL(SchematronResourcePure.java:394)
at com.helger.schematron.AbstractSchematronResource.applySchematronValidationToSVRL(AbstractSchematronResource.java:286)
at org.openpreservation.odf.validation.rules.SchematronRule.check(SchematronRule.java:76)
at org.openpreservation.odf.validation.rules.SchematronRule.check(SchematronRule.java:63)
at org.openpreservation.odf.validation.rules.SchematronRule.check(SchematronRule.java:47)
at org.openpreservation.odf.validation.rules.ProfileImpl.getRulesetMessages(ProfileImpl.java:73)
at org.openpreservation.odf.validation.rules.ProfileImpl.check(ProfileImpl.java:54)
at org.openpreservation.odf.validation.Validator.profile(Validator.java:123)
at org.openpreservation.odf.apps.CliValidator.profilePath(CliValidator.java:87)
at org.openpreservation.odf.apps.CliValidator.call(CliValidator.java:60)
at org.openpreservation.odf.apps.CliValidator.call(CliValidator.java:39)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)

[carlwilson]

I've taken a look at this error now. It's not the same as #262. That's caused by a zip entry that's too large for the file copy/caching step. This is caused by the SAX parser running out of heap space when trying to validate an XML file. The size of the spreadsheet file is related to the problem, but a large .ods file won't necessarily show the problem. It depends on the contents. A large file made up of lots of small sheets or many attached objects, e.g. images, wouldn't show this problem. Your instinct is correct, a large spreadsheet held within the .ods package is the issue, not the size of the package itself. Put another way, not all large .ods files contain large spreadsheets, but all packages that contain a large spreadsheet are relatively large.

I've tried to replicate this problem with some large synthetic files, but without success. When I've artificially limited the heap memory, I get the #262 issue but not this one. I wondered if it might be possible to share an example of a file that exhibits this problem. I'm guessing it's to do with the complexity of the XML file itself. I'm using a SAX parser so that the entire tree isn't loaded into memory, but this will still have to hold some part of the XML tree to allow for parsing and validation.

Another thing you could try is to increase the heap size available to your JVM at startup. This is done by passing an extra parameter to the startup line. As an example, if you wanted to specify six GB of heap memory, this can be done as follows:

java -Xmx6g -jar <path-to-jar> <parameters>

[BjornDNA]

Thanks for details and explenation.
I will send a example of a file that exhibits this issue via other channel. Have not tried to increase the heap size.

[carlwilson]

Hi @BjornDNA #269 goes some way to fixing this as it frees up plenty of heap memory that can be used by the SAX parser. I have been unable to reproduce this. I've tried forcing small heap sizes for my JVM, but I always encounter #262 before the SAX parser runs out of memory. I'll let you know when the new version is available for testing; it will be this week. If you're still encountering this issue, there's new code that will provide me with useful debug information. I'm cautiously optimistic you'll find the next release fixes this.

[BjornDNA]

@carlwilson great! Let me know when there is a release (or is there already?) - then i can test it out here and give you feedback

[carlwilson]

The new release is [here]{https://github.com/openpreserve/odf-validator/releases/tag/v0.18.5) @BjornDNA