Large files result in out of memory error

[AdMSilvan]

During testing ODS files over 50MB have run into an OOM error. Here is an example of an error message:
APP-1: [INFO] Validating C:\Temp\OdsFiles\019846fd-f252-7d85-9970-fd373d693760.ods.
Exception in thread "main" java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large
at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:914)
at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:907)
at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:132)
at java.base/java.io.InputStream.transferTo(InputStream.java:796)
at org.openpreservation.format.zip.ZipFileProcessor.getEntryInputStream(ZipFileProcessor.java:118)
at org.openpreservation.odf.pkg.PackageParserImpl.checkZipEntries(PackageParserImpl.java:138)
at org.openpreservation.odf.pkg.PackageParserImpl.parsePackage(PackageParserImpl.java:110)
at org.openpreservation.odf.pkg.PackageParserImpl.parsePackage(PackageParserImpl.java:81)
at org.openpreservation.odf.validation.ValidatingParserImpl.parsePackage(ValidatingParserImpl.java:77)
at org.openpreservation.odf.validation.Validator.validatePackage(Validator.java:108)
at org.openpreservation.odf.validation.Validator.validate(Validator.java:88)
at org.openpreservation.odf.apps.CliValidator.validatePath(CliValidator.java:74)
at org.openpreservation.odf.apps.CliValidator.call(CliValidator.java:54)
at org.openpreservation.odf.apps.CliValidator.call(CliValidator.java:34)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at picocli.CommandLine.execute(CommandLine.java:2170)
at org.openpreservation.odf.apps.CliValidator.main(CliValidator.java:98)

Thank you for your support with this error!

[carlwilson]

Apologies, I went away on a month-long holiday before I had a chance to look at this. I've taken a look now, and it's definitely fixable. The issue is caching a large file entry. The best fix is to simply not cache items over a particular size. In this case, the file is too big for the maximum Java array. The theoretical maximum Java array size is 2,147,483,647 elements, but this depends on available memory, hence the theoretical. I can replicate a version of the error by starting Java with a reduced heap memory size; it's not quite the same failure, but it fails at the same point. I'll get a fix out shortly and ping this thread when I do.

[carlwilson]

Hi @AdMSilvan, I've put a fix in place for this; see #269, which is much less aggressive about caching large items. I'm just adding the code for extended conformance validation, and I'll put a build up for you to test.

[AdMSilvan]

Hi @carlwilson, thank you very much for the fix! I will forward the information to our team and we look forward to testing the new build.

[AdMSilvan]

We have found that the file which initialy caused the error during testing is still causing the same problem. I can send you the file for testing if that would help. Thank you for your work on this.

[carlwilson]

Hi @AdMSilvan I'm sorry that the problem persists. Sending the file would be very helpful indeed thanks, if you can do so I will take a look as soon as possible. Many thanks

[carlwilson]

Hi @AdMSilvan, I'm struggling to reproduce this with the latest build downloaded from here: https://github.com/openpreserve/odf-validator/releases/tag/v0.18.5

I can reproduce it using the 0.18.4 build. Here's my results at home using the sample you supplied. First the 0.18.4 release:

❯ ./odf-validator-0.18.4-linux corpus/issues/262/019846fd-f252-7d85-9970-fd373d693760.ods
APP-1: [INFO] Validating corpus/issues/262/019846fd-f252-7d85-9970-fd373d693760.ods. 
Exception in thread "main" java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large
        at java.base@17.0.12/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649) 
        at java.base@17.0.12/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
        at java.base@17.0.12/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
        at java.base@17.0.12/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)
        at java.base@17.0.12/java.io.InputStream.transferTo(InputStream.java:783)
        at org.openpreservation.format.zip.ZipFileProcessor.getEntryInputStream(ZipFileProcessor.java:125)
        at org.openpreservation.odf.pkg.PackageParserImpl.checkZipEntries(PackageParserImpl.java:132)
        at org.openpreservation.odf.pkg.PackageParserImpl.parsePackage(PackageParserImpl.java:104)
        at org.openpreservation.odf.pkg.PackageParserImpl.parsePackage(PackageParserImpl.java:81)
        at org.openpreservation.odf.validation.OdfValidatorImpl.validatePackage(OdfValidatorImpl.java:119)
        at org.openpreservation.odf.validation.OdfValidatorImpl.validate(OdfValidatorImpl.java:86)
        at org.openpreservation.odf.apps.CliValidator.validatePath(CliValidator.java:72)
        at org.openpreservation.odf.apps.CliValidator.call(CliValidator.java:58)
        at org.openpreservation.odf.apps.CliValidator.call(CliValidator.java:37)
        at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
        at picocli.CommandLine.execute(CommandLine.java:2170)
        at org.openpreservation.odf.apps.CliValidator.main(CliValidator.java:96)

Next, here's my results for the 0.18.5 release, just standard validation:

❯ ./odf-validator-0.18.5-linux corpus/issues/262/019846fd-f252-7d85-9970-fd373d693760.ods
APP-1: [INFO] Validating corpus/issues/262/019846fd-f252-7d85-9970-fd373d693760.ods. 
APP-4: [INFO] Validation report for corpus/issues/262/019846fd-f252-7d85-9970-fd373d693760.ods. 
DOC-2: [INFO] package OpenDocument version 1.3 detected. 
DOC-3: [INFO] mimetype OpenDocument MIMETYPE application/vnd.oasis.opendocument.spreadsheet detected 
DOC-8: [ERROR] content.xml This XML OpenDocument uses the following extended namespace prefixes calcext. 
DOC-8: [ERROR] styles.xml This XML OpenDocument uses the following extended namespace prefixes loext. 
NOT VALID, 2 errors, 0 warnings and 2 info messages. 

Next, the latest release with extended validation requested:

❯ ./odf-validator-0.18.5-linux -e corpus/issues/262/019846fd-f252-7d85-9970-fd373d693760.ods
APP-1: [INFO] Validating corpus/issues/262/019846fd-f252-7d85-9970-fd373d693760.ods. 
APP-4: [INFO] Validation report for corpus/issues/262/019846fd-f252-7d85-9970-fd373d693760.ods. 
DOC-2: [INFO] package OpenDocument version 1.3 detected.                                    
DOC-3: [INFO] mimetype OpenDocument MIMETYPE application/vnd.oasis.opendocument.spreadsheet detected 
DOC-8: [INFO] content.xml This XML OpenDocument uses the following extended namespace prefixes calcext. 
DOC-8: [INFO] styles.xml This XML OpenDocument uses the following extended namespace prefixes loext. 
VALID, no errors, no warnings and 4 info message found.

It does take an age to validate especially with the preservation profile requested. I'll take a look at performance soon, but as far as I can see the memory issue is fixed. Could you please test again, ensuring that you are using the latest version for sure. If that fails I'll need some more details, specifically:

• OS used and version
• Java virtual machine version and supplier

I'll then test using the same setup to see if that reproduces the problem.