Unable to verify big xml file with StAX (OutOfMemory exception)

[grajsek]

Hi!

With StAX version of signature verification I get OutOfMemory exception. I'm using xmlsec-2.0.8.jar. Exception is thrown at line "inboundXMLSec.processInMessage(xmlStreamReader);" (file: SignatureUtils, method verifyUsingStAX).

XML file size is ~900 MB.

[grajsek]

I'm using IBM WebSphere Application Server v8.5.5.11 Java (Java version = 1.6.0, Java Runtime Version = pwa6460_26sr8fp35ifx-20161110_01 (SR8 FP35), Java Compiler = j9jit26, Java VM name = IBM J9 VM).

Below is image from IBM HeapAnalyzer where you can see the leak suspect

[coheigea]

Probably better to raise this over at Apache Santuario. Is the signed document very highly nested?

[grajsek]

Ok, I will.

Signed document is not highly nested. Below is a sample of the document. Signed part is the element Object with Id="msg" which contains xml with root "Element" which contains alot of sub elements.

<?xml version="1.0" encoding="utf-8"?>
<Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
	<SignedInfo>
		<CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
		<SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
		<Reference URI="#msg">
			<Transforms>
				<Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
			</Transforms>
			<DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
			<DigestValue>...</DigestValue>
		</Reference>
	</SignedInfo>
	<SignatureValue>...</SignatureValue>
	<KeyInfo>
		<X509Data>
			<X509SubjectName>...</X509SubjectName>
			<X509Certificate>...</X509Certificate>
		</X509Data>
	</KeyInfo>
	<Object Id="msg">
		<Element>
			<LotsOfChildren>
			</LotsOfChildren>
		</Element>
	</Object>
</Signature>

[grajsek]

Issue created over at Apache Santuario
https://issues.apache.org/jira/browse/SANTUARIO-466